https://123ash.wordpress.com/feed

After seeing the Github Co-Pilot webage, I got intrigued by the color scheme used to demo the “powers” of Co-pilot. The dark theme caught my attention and I wanted to incorporate into my editor. Unfortunately, there is no official version of the Github Co-pilot theme.

Upon search, I found this add-on https://marketplace.visualstudio.com/items?itemName=BenjaminBenais.copilot-theme which is supposed to provide the same theme. However I notice it did not color my variables and functions in a same way. Now I wondered, how the color of these “functions”, “keywords” and comments is determined.

It turns out, there is a setting in the vscode made specifically for this purpose. It can be set using editor.tokenColorCustomizations. For example, here is the snippet from my setting.

 "editor.tokenColorCustomizations": {
    "comments": "#918d8d",
    "strings": "#226ea3",
    "keywords": "#ee8277",
    "functions": "#6623cb",
    "variables": "#FFFFFF",
    "numbers": "#f78c6c",
    "textMateRules": [
      {
        "scope": "variable.other",
        "settings": {
          "foreground": "#241515"
        }
      },
      {
        "scope": "variable.parameter",
        "settings": {
          "foreground": "#eb4893"
        }
      },
      {
        "scope": "variable.function",
        "settings": {
          "foreground": "#eb7c15",
          "fontStyle": "bold",
        }
      },
      {
        "scope": "Other",
        "settings": {
          "foreground": "#F2AA65",
        }
      },
      {
        "scope": "keyword.operator.other",
        "settings": {
          "foreground": "#F97b58",
        }
      },
      {
        "scope": "comment.line",
        "settings": {
          "foreground": "#918d8d",
          "fontStyle": "italic underline"
        }
      }
    ]
  },

We notice that there is a color setting for comments, strings, keywords, functions, variables, and numbers. I was able to set this but my R code still did not color all the words or now we know it- “Tokens”. The closest wikipedia explanation I found is here: https://en.wikipedia.org/wiki/Lexical_analysis#Token.

It turns out those missing “token” can be set using “textMateRules”.

 {
 "scope": "variable.other",
 "settings": {
      "foreground": "#241515"
   }

Basically, it required “scope” property for the target token and settings for color and font option. Please note that we still cannot set font.size property for these tokens. How to find what are the names of the tokens in your code. There is a inbuilt way within vscode to find out using (cmd+shift+p) and typing “Developer: Inspect Editor Tokens and Scopes”. Once activated, hover your mouse our target words to find the names/scopes of these tokens.

We see that main belongs to “support.function.go”. We can use either “support.function.go” or just “support.function” to use it in the textmate rules setting. That is it. It should now work with your color choice and font style.

Centered log ratio is a very common approach when one needs to transform the data which is skewed such as composition data or some other count data.

In order to perform centered log ratio transform or CLR transform, we need to use the concept of Geometric Mean. Geometric mean is one of the oldest method to take the mean of the sequence of numbers and it extends beyond that but we won’t go that far. One of the cool things I found in the Wiki link for Geometric Mean is the that

area of radius of the circle is the geometric mean of the semi-major and the semi-minor axes of the ellipse formed by deforming the circle

π r2 = π*a*b

Here, the circle is formed by deforming the ellipse. This tells us that circle is the special case of ellipse where semi-major and semi-minor axes happens to be of same length. Now if we simplify, we get

r = √a*b

Now remember, here we have two numbers a and b and that is why we take square root. If we have n numbers, we have to take nth root. This nicely leads to the geometric mean formula. Let us fire up R terminal and type this code.

x1 <- c(0,1,2,3)
geometric_mean = (1*2*3)^(1/3)

If you notice we ignored zero because if we multiple any number by zero, we get zero. If we want to abstract the geometric mean formula for sequence of numbers beyond 3, we can write it as

So in English, we can say that is the nth root of the product of all numbers where n is the count of numbers.

The other main component for centered log ratio is natural log. This natural log comes from taking inverse of exponential function where the base happens to be e.

ex

ln(e)x= x (Meaning that this ln() is the inverse of the e. Read more about e at this link from purplemath.com . In addition, I also found this link at plus.maths.org was very helpful. In short it is the related to the upper limit of the growth rate. It has important implications in almost every field.

Now that we have some idea about natural log, the formula of CLR is:

ln(x1)/g(x), where x1 is the one element it the series of numbers or vectors of numbers and g(x) is the geometric mean of the complete vector or series of numbers.

Using R, we can go back to previous number series.

x1 <- c(0,1,2,3)
geometric_mean = (1*2*3)^(1/3)
log(x1/geometric_mean)

> log(x1/geometric_mean)
[1] -Inf -0.59725316  0.09589402  0.50135913

This is correct but we need to get rid of zero in addition we also need to prevent the log and geometric mean function from taking taking zero into account.

Let us fix geometric mean function so that it ignores zero.

geoMean <- function(x) {
x1 <- x[x>0] # Get rid of any zeroes in the list
return (prod(x1)^(1/length(x1))) #prod is an inbuilt function
}

Now there are other formulas that can calculate geometric mean but I think this function is more true to the mathematical formula as shown above. Our idea is to build intuition.

So now we can build our custom clr function:

x1 <- c(0,1,2,3)
geoMean <- function(x) {
x1 <- x[x>0] # Get rid of any zeroes in the list
return (prod(x1)^(1/length(x1))) #prod is an inbuilt function
}

# Our custom CLR function
ourCLR <- function(x) {
x[x>0] <- log(x[x>0]/geoMean(x))
return(x)
}

We have used logical subsetting to only replace values greater than 0 and this also allows us to get the original list with all the elements replaced with transformed values along with the zeroes as it is. If we do not use the x[x>0], we will get the transformed list but with zeroes left out. We want to retain the zeroes otherwise we get truncated list.

> ourCLR (x1)
[1]  0.0000000 -0.5972532 -0.8270395  0.8270395

CLR transform is “isometric” in the sense that it preserves the distance among the elements. Now that is the topic for another blog post. Thanks!

These are some of the code snippets that I often need while using ggplot2 library.

1. Increasing the font size of text to around 20
2. Rotating the labels so they are visible in case they are very long

library(tidyverse)


mtcars %>%
  ggplot(aes(x = reorder(row.names(mtcars), mpg), y = mpg, fill = mpg)) +
  geom_point() +
  theme_bw() +
  theme(text = element_text(size = 20)) +
  theme(axis.text.x = element_text(angle = 90, vjust = 0.5, hjust = 1))

library(tidyverse)

After hearing about ARM macs in November 2020, it was about time to test one myself. I heard lot about overwhelmingly positive experience with everyone. I was reluctant to try one since I already have the current 18 wheeler of the Mac computers – Mac Pro with 12 cores and RAM upgraded to 240GB (Plenty more than I need on day-to-day basis). There was no need for me to try it out. However, I kept eyeing the refurbished section to see if they have any cheap Mac Minis. Moreover, I had some trouble with Pro Display XDR and it turns out Mac Pro is the only computer I have that can be used with it.

Finally, I saw Macmini listed on the refurbished site with all souped up specs (16 GB RAM and 2 TB drive). 10GB ethernet would have been nice but they are just coming into market so not available on the refurbished market at the time of this writing. Nevertheless, this is the powerful M1 MacMini I was able to get.

First Impression:

This machine is as fast as my 2019 Mac Pro in day to day usage. I mean literally I did not feel any slowdown in multi-tasking involving browsing web, coding R, reading PDFs, PowerPoint, Word and Excel simultaneously. It did swap a lot but nothing I could notice in my usage. I guess having 2TB SSD definitely helps with the swapping as it has higher read speed. If you go to the Apple’s website, all their claims about speed are based on the MacMini with 2TB SSD.

However, one of the best features of MacMini I feel is the lack of heat and noise. Literally the temps stayed about 34 ºC and 52º C at one of the GPU cores but it is so cool to touch and did not show any signs of struggle in computing whatever I threw at it. It feels almost as if someone took iPad internals and converted to a full blown desktop. We know that it was born out of their CPU in iOS devices. This was not the question of If but when. It is funny when you see that Apple advertises that M1 is coming to iPad when it was other way around.

Now to some of the issues that can be annoying but nothing critical when replacing Mac Pro with M1 MacMini.

• Fewer Thunderbolt ports : I ran out of Thunderbolt ports if it was not for my Pro Display XDR acting as the hub. Adding non-Apple wireless keyboard with receivers will be fill up those USB ports very easily. I think if using Mac Mini as the primary computer, one needs to be an hub extension. Also, I was using one of the the thunderbolt port for connecting with Mac Pro since the network connection with 1GB port will be lot slower (This won’t be an issue with the updated 10Gb port ethernet)
• Lack of physical RAM: Although I never felt that I was RAM, I think it is matter of time if I have to use more application where RAM will be an issue. Lot of bioinformatics program need lot of RAM and can suck CPU usage in no time. I also have not tested running Rstudio program which can slow the computer if kept running for long time ( Memory leak anyone?).
• Glitches with the display after coming from sleep: I found that sometimes the display would be glitchy for about a second and then it would be OK. This is a very minor issue but I thought I should mention it.
• Raid storage: Promise pegasus thunderbolt raid storage drivers installation would require kernel extensions change by rebooting the computer and setting security to “Reduced security”. Not sure if it practically reduces the security. I had to similar thing for Audio Hijack installation. I hope this is a temporary solution and having end-users change those setting is not very ideal.

Unified memory architecture: This is one of the most interesting aspect when one talks about M1 processor architecture. If you are curious about the speed of the RAM, it may be around ~4266 MhZ range according to this reddit post: https://www.reddit.com/r/mac/comments/kmwmho/ram_speeds_on_m1/ This is very faster than current generation of Intel Macs which are about 2666 MHZ. This may not be a very appropriate comparison but it gives rough idea. The idea of having RAM sharing between CPU and GPU is not dissimilar to idea of having integrated graphic with Intel but from what I gather, it seems the RAM is “closer”, more tightly integrated, and has higher bus speed as compare to traditional integrated graphics solution such as Intel Iris. It means it will be faster but also not upgradable post purchase.

After spending few days, I decided to give this Mac Mini back. It is a nice computer and probably very capable for doing any common task. However, the limitation of RAM if I needed to run memory hungry bioinformatics program swayed me to return it. Also, my Mac Pro 2019 is running as solid as ever but using M1 Mac Mini, I barely noticed any difference which is just amazing. Very low heat, barely any noise when running most common tasks makes this win win computer.

My usual interest in the WWDC has waned over the past four years. I remember being so excited about WWDC for iOS updates. Granted things were improving or new features were introduced at faster rate. May be my expectations have reached levels that exceeds the current improvements in iOS. I can say the same for MacOS but I will leave it for another post. It definitely felt it was pre-recorded. Being at home, I watched WWDC anyways and some of the things got me interested.

One of my favorite feature was the inline reply to specific message. Using Whatsapp has made be realized the shortcomings of iMessage. I mean this has be so frustrating to follow the context because if you miss replying to a message, it might not make sense at all. My personal use case does not involve buckets of messages but I am so much looking forward for this feature.

App library is the other feature that I think is worth upgrading to the iOS 14. If you have lot of Apps that you use infrequently, it make it hard to browse. Granted you can make folders but if you are like me and avoid making folder, this feature is seemingly a good solution. Moreover, having folder that are automatically curated, it is easier to delete or remove them. iOS 14 can also not show them or make them invisible. It is reminiscent of features from Windows 7 which allows you to “Group” items together regardless of their location.

This feature is the indicator of how far we come since the introduction of Apps. Apps being the center of our Smartphone use, we have moved to the more service mode where we pay for the services and Apps are just the medium. There are games and other fun apps but apps related to ride-sharing, music, and news takes centerfold. This is already old news, but in the context of viewing and browsing apps, it makes lot more sense to have some automated organization.

Another related feature is the App clips. This feature is almost having the advantage of App world without having physically installed on your phone. It is limited to 10MB of space. I wonder what it actually means. There are so many apps we install and rarely touched again. It reminds me of Apple pass which we can use for holding boarding pass and insurance cards. My concerns are will App clips work with limited connectivity. 5G and WiFi can allay such concerns but we know enough that network is always so brittle even for 10Mb. I am intrigued nonetheless.

App store privacy information and Safari Website Privacy Report are welcome changes. At this point, it is almost expected to be tracked over the websites by someway or another. I am not sure if consumers are interested in knowing all the details about cross-site tracking. Another notable change is the indicator for Microphone recording and notification for clipboard access. I felt Apple wants to strengthen the privacy oriented vibe. Regardless, as a tech enthusiast, I look forward to these changes.

I am not going to reflect over other features such as widgets, banner phone call notifications, Translate and Maps which I think are welcome improvements and long overdue. Overall, I feel iOS 14 is going to be good.

After my iMac pro debacle, I decided that it was a time to switch to proper workstation type tower computer. I think iMac pro is a great computer but the convenience of having all in one computer hurts when something goes wrong with it. Upgrades are also very difficult in iMac pro. Opening iMac pro is virtually impossible without experience. Moreover, it can be confined by the physics of cooling and is throttled in terms of sustainable CPU and GPU speed. All this lead me to decide about moving to a classic modular desktop computer : Monitor + CPU.

It has been more than 10 years since I purchased a tower computer. Mac Pro just got released and initially I was not even planning to buy a new computer. However, my fresh experience with iMac pro virtually pushed me to the edge to buy this expensive and giant computer. Ordering Mac pro was very difficult process. Obviously price is the most important factor but the cheapest option had some low ceiling in terms of processor speed and RAM speed. I ended up buying the base model with processor upgraded to 12 cores and storage upgraded to 1TB. I will go with all the options on the Apple website and explain my reasoning about my choices for this machine https://www.apple.com/shop/buy-mac/mac-pro/tower#

The main reason to select 12-core processor vs 8-core processor is the ability to use RAM at marginally higher speed at 2933 MHz up from 2666 MHz. Whether this is makes difference in practice is another matter. In any case, if one needs higher core, there is not much choice anyways. I chose 12 cores which I think was worth $1000 extra. CPU is upgradeable but not as easy as other components. There are plenty of examples, people upgrading their CPU in previous generations Mac Pros. I prefer not to mess with it.

Talking about the Memory, I chose not to upgrade from 32 GB RAM which is the whole point of this computer i.e. to upgrade as needed. I do not need much RAM in day-to-day use but having more RAM is future proof . RAM helps to run thing smoothly by using that “cache” reserve.

Interesting range of options here but no options for 64GB which is what I have in iMac pro. Since Mac pro has six memory channels, the memory has to be multiple of 6. One can see that base model option is the only option that does not follow this rule. However Apple provides a good guide to upgrade RAM at this link : https://support.apple.com/en-us/HT210103

I am planning on upgrading my Ram to minimum 96 GB which is enough to run occasional metagenomic analysis. It should also provide plenty of cache if needed. My prior experience of having 64 GB of ram showed me that I rarely used that much RAM. Is seems waste but iMac Pro is not easy to upgrade while Mac Pro can be upgraded easily. Needless to say, RAM is the most easiest and gives you more bang for buck as compare to other upgrades.

Upgrading RAM on Mac Pro is not anymore difficult than on any PC. In an ideal world Apple would let you upgrade RAM at not very high prices but at the time of this writing it will cost $1200 to add 64GB (2*32) which seems excessive if you decided to not upgrade to 96GB at time of purchase. The same configuration would cost you $1000 more at time of purchase. Essentially saving you $200 if you are adamant on buying RAM from Apple. These RAM are made by either Hynix (which Apple seems to prefer), Samsung, or Micron (available from Apple for 64GB and above option). My limited experience of upgrading stock experience has not been great (n=1). My 2007 MacBook Pro did not survive long enough after the upgrade to OWC RAM (Samsung). It could be just a coincidence.

Anyways, I upgraded RAM to 96GB by buying the aftermarket Micron RAM from memory.net (https://memory.net/product/mta36asf4g72pz-2g9-micron-1x-32gb-ddr4-2933-rdimm-pc4-23466r-dual-rank-x4-module/) . I wanted the Hynix ram which comes with this Mac pro but the Memory.net reported that they were out of it so offered me Samsung brand. I hesitated since I wanted to have the same manufacturer for all the slots. Looking closely at the Apple RAM upgrade options with 64GB and above, I found they were all Micron instead of Hynix. So I decided to go with Micron RAM instead of Samsung. The upgrade process was easy but I was confused by the orientation the RAM cover should be put back. I followed the RAM upgrade procedure as described here : https://support.apple.com/en-us/HT210103 and I closed the lid but the boot light showed orange light which probably meant something did not went right with the RAM upgrade. I had to open the lid again and had to make sure the RAM are snuggly fit. It worked the second time!

This was probably the easiest choice for me since I am not running graphics intensive process yet. This feels like a downgrade coming from Vega 56/64 HBM2 GPUs but practically I should not feel any difference in day to day use. I would add that GDDR6 options looks enticing.

I believe Apple should have been generous on this option considering the boot OS drive is tied with the T2 security chip. As described on this page https://support.apple.com/en-us/HT210556, upgrading boot drive to higher storage will required visit to Apple service provider. With my recent experience of hauling iMac pro, I am not looking forward to this experience. With all this in mind, I went with 1TB upgrade. It should be enough to hold the OS updates and Apps. I feel like storage should be selected at-least twice the amount of you think you use. To do some cost savings, I went with 1TB. I may regret this option but we will see.

Again this option was not really important for me. Although, I must mention that this is an FPGA ( Field Programmable Gate Array). In other words, reconfigurable hardware. I absolutely have no need for this yet but I am looking forward to Apps that may use it. This is a very future looking accessories if you are not into video processing yet.

Wheels that $400 is relatively expensive. It makes it easy to move this gorgeous computer but then I thought how often I need to move this thing. I think $400 is not worth the luxury of wheels. Therefore, no wheels for me.

If you used Apple Mouse and Keyboards, you will immediately be drawn by the silver underneath of both the mouse and trackpad. I have used the first trackpad version but it never caught up with me. I decided to go with the plain old Magic Mouse 2 and US keyboard.

This is again something I do not need. No need at the moment for any of these software for any video and/or audio processing.

I must say that the power cord and USB-C cable feel very nice and do feel like these braided cables can withstand years of use. I mean for $6000 it screams for such cables. I do hope that could last longer than the one included with iMac lines. I wish keyboard and mouse also showed some different touch apart from the color as they did for the cables.

My main use cases are running data analysis and plotting figures mostly using R-studio IDE. This does not put this machine to any stress at all. It is way overpowered for that purpose. The real test comes with the software to analyze genomic data. I have plenty of raw data from Microbiome sequencing reads. This is an ideal data to put the CPU under stress for longer period of time.

I happened to have Soil microbiome dataset that I needed to analyze for protein content. I decided to use https://github.com/bbuchfink/diamond for that purpose. It is faster than NCBI blast software and practical enough to be used on workstation. Overall, the file input size is around 11 GB gzipped. I decided to use 9 CPUs out of 12-cores in order to spare some CPU cores for other side tasks such as web-surfing and playing music.

Here are some stats from the Intel Power Gadget software sampled for 13 minutes (https://software.intel.com/en-us/articles/intel-power-gadget) which I saved on CSV file https://drive.google.com/file/d/1DETIWe6V4Ub1_AKO53C1igJFogzzofxw/view.

If you just want to get gist of it, the CPU core temperature ranged from 92 – 70 degree celsius! (197.6 – 158 degree F). Now that is a very hot temperature. The 3 huge intake fans in the front were at 500 RPM. Average CPU (Xeon W-3235 3.3 GHz 12-core) power usage was at 162.87 Watt and DRAM (96 GB (8GB*4)+(32GB*2)) power usage was at 41.75 Watt. The thermal output for Mac Pro as per Apple website is here : https://support.apple.com/en-us/HT201796

Here is the snapshot of temperature using this command below.

sudo powermeterics

The ambient temperature using my infrared gun were 29-31º C in top-front side and 43-45 º C in the top back. I could feel the heat emanating under my desk. I wish it was bit cooler outside so I could get the warm fuzzy feeling from my desk. For the reference, the ambient temperature was 26.67 ºC (80ºF) in the room. The sound level as measured by the 4th gen Apple watch is around 34 dB. I can barely hear it behind the background noise in my home. The process has been running for close to 24 hours and the temperature have stayed remarkably at this level.

I am not sure what can I say about the longevity of this machine based on this informal test. It seems to be really silent and I love silent machines.

Imac pro has given me the most trouble in the history of my computer usage. While the first repair one month, the other repair took over 5 days. The reason for my second repair was the smudge in the lower right corner of my screen. That repair was faster because iMac pro uses the same screen component as the regular 5k iMac.

However, it worries me that my computer stays warm after going to sleep. It seems very irregular since I checked it in the morning and it was just around 32℃. However, I came back in the evening and it felt really warm around the computer. I could not hear any fan noise. I measured the temperature with an infrared temperature gun and it was over 37℃.

I immediately started the computer and the temperature went down pretty fast. As I type, the temperature is around 30℃. It also feels cool to touch since the infrared temperature gun only tells us about surface temperature and not the temperature inside the computer.

I searched the web for some suggestions:

I noticed this on my replacement iMac Pro…my first one had a problem with panic attacks and restarting…my replacement runs fine but after 2weeks noticed that once awakened sometimes the fan would blow for a short period. using istat the temperature would rise overnight in sleep…I could track this consistently….and once awake and even heavy use it would cool down (the cooling system worked)…to kinda solve this issue I had to check “prevent this computer from sleeping automatically when the display is off”…..I think in sleep the fans shut off but the work class components are generating heat.

https://discussions.apple.com/thread/8321714

There was some suggestions to check the pmset log using this command:

pmset -g log|grep -e " Sleep  " -e " Wake  "

There was another suggestion to check the logs at this location:

/var/log/powermanagement

Another log to check out is:

log show | grep 'Wake reason'

I am still investigating this issue and I am hoping this DOES NOT turn out to be 3rd repair for this iMac pro.

Also there is also another command to find what is preventing mac from sleeping.

pmset -g assertions

Update: 7th March 2019

So it turns that my iMac Pro is working fine after SMC reset. I keeping eye on the temperatures using iStats Menu. I have not restarted it since then and I hope it continues to work after a hard reboot.

Update: 20th October 2019

Unfortunately, the issue cropped up again after update to Catalina. It started shutting down with no reason and I took it to the repairs. Got it back from the store saying that issue is software related. However, I noticed the dark spots on the bottom right corner which the technicians had not fix. The store eventually took it for repair again and I got it back in 2 days.

Update: 2nd November 2019

The random shutdown started again and I talked with Apple support. The support decided to get me replacement since it is the Apple policy to replace computer if there are more than 3 incidents within the year. The support person gave me two option whether to replace it via store or directly through apple. I asked which would be the faster option, the support replied that in-store replacement would be fastest. Also, they quoted me it would take six weeks to get the computer back and that it will be ordered by the store.

I waited for two weeks and I found that the store had not ordered it. After physically going to store for the update, I found that nothing had been done. They finally did something. Apparently, because my iMac pro is custom order or not available in store, it would take longer.

Update : 12th December 2019

I finally received a new iMac pro. It took a while but Apple gave me a replacement. I am so glad. The new computer has not shown any of signs found in my old iMac pro.

By CDF, I mean Cumulative distribution function and by PDF, I mean probability density function. These are related to each other and are very important in understanding statistics and probability.

I came across these terms while I was plotting a histogram of temperatures in Houston and Chicago. Normally, when we have some data, we can plot the histogram using simple hist function. For example,

#Load library
library(tidyverse)

#Read the Houston weather data
houston_weather <- read_delim("KIAH_daily.txt", delim = ",")

#Read the Chicago weather data
chicago_weather <- read_csv("KORD_daily.txt") 

houston_weather %>%
  bind_rows(., chicago_weather) %>%
  mutate(Date = as.Date(Date, format = '%m/%d/%Y')) %>%
  separate(Date, c("year","month","day")) %>%
  ggplot(., aes(x=`Avg Temp`, fill=Site4, color=Site4)) +
  geom_histogram(position="identity",  alpha = 0.5)
  

I was wondering if there was a better way to represent this information and this blog (http://www.ericmjl.com/blog/2018/7/14/ecdfs/) caught my attention. It talks about how ecdf are better representation for showing the distribution and the outliers. However, I was having difficulty understanding it until I saw this blog post (https://blog.demofox.org/2017/08/05/generating-random-numbers-from-a-specific-distribution-by-inverting-the-cdf/) where it says that

A CDF is a function y=f(x) where y is the probability of the number x, or any lower number, being chosen at random from that distribution.

https://blog.demofox.org/2017/08/05/generating-random-numbers-from-a-specific-distribution-by-inverting-the-cdf/

Let us plot it to understand what the above definition means.

houston_weather %>%
  bind_rows(., chicago_weather) %>%
  mutate(Date = as.Date(Date, format = '%m/%d/%Y')) %>%
  separate(Date, c("year","month","day")) %>%
  ggplot(.,aes(`Avg Temp`, color = Site4)) +
  stat_ecdf(geom = "point") 

If we read that function definition, this figure tells us that given this data, there is 50% or 0.50 in Houston probability that temperature will be at ~ 72℉ or less (Read where the axis is at 72 ℉ and find the corresponding y-axis value). Similarly, we can say that there is 50% probability of finding temperature of 50℉ or less at Chicago.

This was really easy to understand but there is a caveat, here we are talking about ECDF “Empirical” CDF. This means this is not a representation of the entire set (or theoretical distribution) but what was observed and hence the word “empirical”. Please read this post: https://stats.stackexchange.com/questions/239937/empirical-cdf-vs-cdf

The related term is PDF (Probability density function). What does it tell us? let us plot the same data as above using probability density function.

houston_weather %>%
  bind_rows(., chicago_weather) %>%
  mutate(Date = as.Date(Date, format = '%m/%d/%Y')) %>%
  separate(Date, c("year","month","day")) %>%
  ggplot(.,aes(`Avg Temp`, color = Site4)) +
  stat_density(geom = "point", kernel = "gaussian", n=1024, alpha = 0.6) 

This is just a smoothed histogram and the smoothing was implemented using a Gaussian kernel. Basically, we are smoothing (estimating) between the actual values using a function. This whole curve (made from the consensus of the multiple curves) is normalized so that the area under the curve equals one.

PDF is essentially a derivative of CDF so it will always be positive since CDF is always monotonically non-decreasing (never goes down but can go up or stay steady).

https://www.che.utah.edu/~tony/course/material/Statistics/18_rv_pdf_cdf.php

For any continuous variable, we will never get the exact probability of measuring that number. For example, since temperature is a continuous variable, we can never tell about finding “a particular” temperature or saying that the probability of getting 70℉ is some x percent. That probability will be zero. However, we can say that the probability of finding the temperature in the range of 69.9 to 70.02 is some x percent. There will be always some infinitely small uncertainty.

We can also play with n parameter in the stat_density which tells the code to use that many points to determine the curve.

Read more about Kernel density at these links:

https://seaborn.pydata.org/tutorial/distributions.html#kernel-density-estimation

https://stats.stackexchange.com/questions/4220/can-a-probability-distribution-value-exceeding-1-be-ok

https://stats.stackexchange.com/questions/48109/what-does-the-y-axis-in-a-kernel-density-plot-mean

This is the update on the previous posts. Please see iMac pro repair 1 and iMac pro repair 2 for the background.

Actually, there is a good news that I finally received my iMac pro after ~ one month. The problem started on 24th December 2018 and I received it back on 27th January 2019. I was notified by the email that my computer is ready for pick-up late night yesterday.

The check out was smooth and I noticed that they replaced my logic board with a more advanced version of the graphics card (Vega 56 to Vega 64). This was totally unexpected in a very good way. Granted they made me wait for one month but the wait was worth the upgraded graphics card

I was told last Monday that they will fix it in around 5 days once they receive the part. The part ( logic board) took forever to come. It was not delivered when they first ordered it and had to order it again. This was strange but I was more worried about the unsuccessful repair process.

Fortunately, it was repaired in the first attempt. So far my computer seems to function as expected. My faith in the Apple INC has been restored.