https://techsavvypriya.wordpress.com/feed

Here are tips to learn coding in a more efficient way. They are backed by research in Neuroscience, Behavioral Psychology, and over a decade of experience.

You can spend months going to the gym and still see zero results because of how you work out.

Most of us know this.

The same thing applies to coding.

You can spend months learning, trying different things, jumping around tutorials, and still feel like you’re a beginner – at best, an advanced beginner.

And it hurts.

This makes you wonder if there’s a way to learn faster, better, more efficiently.

There is.

But first things first.

“Speed will follow when the mechanism of the movements is more assured.”
― Rafael Sabatini

If you don’t understand the fundamentals of learning, it doesn’t matter how fast you go, you’re still building on a shaky foundation. And you don’t want that.

So with that, let’s get to it.

The 2-minute tutorials

If I told you there was a 2-minute tutorial to help you master JavaScript right away, would you check it out?

Whether you answered “yes” or “no” , it doesn’t matter.

That is, provided you understand deep down that a short tutorial alone goes against the principle of mastering any domain.

Here’s how mastery works in a nutshell:

1. Start with a huge information dump.
2. Compare new learnings to what you already know before.
3. List out the ways that old and new info differ.
4. Test yourself by creating your learnings from scratch.
5. And finally, teach someone else.

This. Takes. Years.

That’s why shortcuts alone won’t cut it.

So the next time you see a once-in-a-lifetime tutorial promising you mastery in one fell swoop, remember that at best it is merely giving you information dump.

Internalize this and your foundation will be solid.

Then speed will follow.

How do you deal with struggling?

I’ve seen people quit programming because they don’t want to struggle with bugs.

That’s like hating the gym because you don’t want your workouts to hurt.

But the fact is this: you don’t run from bugs – you train yourself to find them quicker.

Because a proficient programmer is simply an efficient detective of bugs.

Every time you feel frustrated by bugs, remember that this is where a learning opportunity opens up. Don’t skip it.

“Toleration is the greatest gift of the mind; it requires the same effort of the brain that it takes to balance oneself on a bicycle.”― Helen Keller

Neuroscience research confirms that practice doesn’t simply make you perfect – it is deliberate practice to improve in a specific task that makes you better.

And deliberate practice involves struggling. With bugs.

So your expectation shouldn’t be to code flawlessly.

Rather, your mantra should be:

• You code.
• You debug.
• You struggle.
• You overcome.
• And you repeat.

The earlier you embrace this, the more efficient your learning becomes.

So if at this point you’ve not struggled at all, you’re simply playing around. And wasting time.

But if on the other hand you feel like you struggle every single time, then the next section is for you.

You’re not dumb, you’re learning at the wrong level

Let’s start with a little analogy.

Imagine learning the English language (the language of exceptions).

You start with a tutorial motivating you to learn to spell big words like ELEPHANT.

But it’s not working.

After beating yourself for so long you decide to go to a lower tutorial and learn to spell smaller words like ANT.

But then you still struggle.

So you conclude that you’re dumb because you lowered your tutorial level and still didn’t grasp anything. You have proof – negative proof.

But when a new teacher investigates why you’re struggling, they find out that you’ve been struggling because…

You actually don’t know the English alphabet.

Let’s bring this back to coding.

Some people jump into beginner tutorials to immediately start learning React or Angular (ELEPHANT).

Then realize it makes no sense so they move on to learn JavaScript (ANT) as introduction to the world of programming

But that still makes no sense to them, so they quit.

“I’m so done with learning to code. This is clearly not for me.”

But they’re simply working with something that’s too advanced.

Even introduction courses can be advanced because your thinking is not yet wired to programming in general.

I don’t see this recommended enough, but if you’re struggling with intro courses, you should consider Intro to programming for kids.

There’s no shame in finding something lower than intro courses and starting from there to build up.

Learn with tools like:

• Alice.
• Scratch.
• Codewars.

Since you’re older, you’ll likely go through them faster.

But they’ll give you the much needed soft intro that makes sense and builds your appetite for programming.

Again, this has nothing to do with ego.

It’s your learning journey.

If you learn the alphabet first:

• You’ll gain the confidence needed to start spelling ANT.
• Then ELEPHANT, as your next big step.
• And some day long words like pneumonoultramicroscopicsilicovolcanoconiosis.

It doesn’t matter what discipline or area of coding you jump into, this holds for Intro to Python as much as it holds for Intro to Machine Learning.

You’ll learn more efficiently if you learn at the right level.

Elevate how you learn with spaced repetition

This technique is from research in Neuroscience. I’ll describe it quickly.

How it works:

• You learn.
• Recall in 48 hours.
• Then review in 72 hours.

Doing this:

• You’ll forget less.
• Connect new ideas to old, and;
• Retain more in long term memory.

I put 48 and 72 hours as an example, but you can expand that timeline to days and weeks. The most important takeaway is not to simply learn once.

But to intentionally set aside time for you to come back and review.

The longer the delay between reviews, the more it gets buried in long-term memory.

• Duolingo uses it.
• Quizlet adopted it.
• You can also apply it.

1 project > 10 hello worlds

Don’t get caught up in hello worlds forever.

I understand the thrill of completing simple exercises:

print “I am awesome”
print “You’re cool”
print “Wait, are we cool?” 
print “OMG! Guys! Like I can so print all day” 

But I have to be honest with you, this won’t cut it.

Going back to our ANT analogy, no matter how many ways you rearrange the words ANT, you’ll never arrive at ELEPHANT.

No matter how many introduction tutorials you learn, you’ll never really get past the basics unless you push yourself beyond that.

“But I don’t know what projects to start as a beginner.”

No worries.

Here are 3 examples:

• Currency converter: Create a program that converts currencies from one unit to another, for example, converting Indian rupees into dollars, pounds to euros, and so on.
• Random Password Generator: Create a program that takes some words from the user and generates a random password using those words.
• Guess the number: Ask a user to guess a number between 1 and 50. If they guess outside that range, show an error message asking them to guess again. Whenever they guess the wrong number, ask if they want to keep playing or they’d like to quit. Finally, when the user eventually guesses the right number, congratulate them and show the number of attempts they had.

And if all of them seem too difficult or boring, message me on Twitter and I’ll create something tailored to your level.

Remember, this is not about ego.

It’s about your learning level and growth.

Motivation is like taking a hot shower

Taking a hot shower feels good – you should do it.

But if your hot water is out for 2 weeks, would you not shower?

The question sounds ridiculous.

But in the same way, you shouldn’t base your learning on only days when you’re motivated.

There are lots of exciting days in coding, but there will also be dark, de-motivating days.

Honestly, this is not just with coding. It’s with life and embarking on any new adventure.

But you have to train yourself to show up not only when you’re feeling hyper inspired, but also on low, boring days.

This doesn’t mean forget motivation entirely.

Far from it. Here’s what you should do instead:

On days when you feel super motivated, use your motivation to quickly commit yourself to something big that will keep you accountable when the motivation goes dry.

How?

Join an accountability movement like:

• 30daysofcode
• 100DaysOfCode

Something.

Anything.

To keep you accountable on slow days.

In Behavioral Psychology, this is known as pre-commitment.

You use moments when you’re optimistic to commit yourself in the future for when you have zero motivation. Use this technique and you’ll keep yourself going for a longer period of time.

The journey of learning is never-ending

No one really teaches you how to learn.

The focus is mostly on what to learn. But learning how to learn is crucial if you want to go far.

You may have started with the idea of simply learning to code or to one day become a developer.

But there’s more.

When you come out on the other side, you’ll come out not just a developer.

But as a renewed person because you’ve picked up a new way of learning and approaching things.

Don’t get me wrong.

This isn’t easy peasy lemon squeezy – you’ll work like hell.

But in the end, it’ll be worth it not just because of coding but because of the joy of transforming yourself.

Thanks for reading!

And that’s a wrap. Not going to lie, this took a lot of hours and reflection to put together.

If you like this please buy a coffee for me

buymeacoffee.com/?via=priyapareek

SQL statements: DDL, DQL, DML, DCL, TCL.

SQL-Structured Query Language is a standard language for storing, manipulating and retrieving data from databases.

SQL statements are categorized into

1. DDL (Data Definition Language),
2. DQL (Data Query Language),
3. DML (Data Manipulation Language),
4. DCL (Data Control Language),
5. TCL (Transaction Control Language),

Sometimes, Select query which we categorized as DQL may be considered in DML.

DDL

Data Definition Language is used to define database schema. Also, to create and modify the structure of database(db).

Database schema means a skeleton / structure which represents logical view of the entire database.

Create

To create database or its objects (tables, index, functions, view, triggers, stored procedures).

For database : create database db_name;

for table : create table table_name (column_name datatype);

Alter

To modify the structure of already existing table.

Syntax : alter table table_name add column column_name datatype;

Change : To change the name of column, column data type, column constraints.

Syntax : alter table table_name change column column_name datatype constraints;

Modify : To modify existing column data type and constraints.

Syntax : alter table table_name modify column_name datatype constraints;

After : To add new column at specific position in the table.

Syntax : alter table table_name add column_name datatype after column_name;

Drop

To delete an existing db or its objects.

No where clause is used to drop.

Drop command can’t be rollbacked because works directly on data itself.

Syntax : drop database db_name;

Created a dummy database named db_name and deleted using drop.

Syntax : drop table tb_name;

Created a dummy db named as db_name and created a table tb_name then delete by drop table.

Drop table helps to remove a table definition and all the data, indexes, triggers, constraints and permission specifications.

Truncate

To remove all records (row) from the table but the table’s schema wont deleted.

Syntax : truncate table tb_name;

After truncate, use desc truncated_table_name to see the schema. If you need to delete that schema also then use drop table table_name.

Rename

To rename an existing table name.

Syntax : rename table existing_table_name to new_table_name;

DQL

Data Query Language is used to retrieve data from database.

Syntax : select column_name(s) from table_name <where condition> ;

To retrieve all data : select * from table_name <condition>;

To retrieve specific data : select column_name1,column_name2 from table_name <where condition>;

DML

Data Manipulation Language is used to modify the database by inserting, updating, deleting the data.

Insert

Inserting data into the table of a database.

Syntax : insert into table_name (col1, col2,.., coln) values (v1, v2,..,vn);

We can insert values without mentioning the column name of the table.
Syntax : insert into table_name values (v1, v2,…,vn);

Update

To update / modify value of column in the table.

Syntax : update table_name set col1 = v1,…, col_n = vn where condition;

Delete

To remove one or more rows from a table .

Syntax : delete from table_name where condition ;

DCL

Data Control Language is used to give rights and permissions to the users of the database.

We cannot modify our internal marks displayed in the college result portal but our faculty can do. This is because of rights and permission given by the admin of that portal (college exam cell / university authority).

DCL : Grant and Revoke .

Grant

To give access privilege to database.

Syntax : GRANT privilege_name ON object TO user;

• privileges_name : the access rights or privileges granted to the user.
• object : the name of the database object to which permissions are being granted. If we are granting privileges on a table, then table name will be the object.
• user : the name of the user to whom the privileges would be granted.

Privilege

Global : GRANT SELECT ON *.* TO user1@localhost;

user1@localhost can query data from all tables in all database of the current MySQL Server. (to denote all we use * )

Database : GRANT INSERT ON db_name.* TO user1@localhost;

user1@localhost can insert data into all tables in the db_name database.

Table : GRANT DELETE ON db_name.table_name TO user1@localhost;

user1@localhost can delete rows from the table table_name in the database db_name.

Column : GRANT SELECT (column_name(s)), UPDATE (column_name1) ON table_name TO user1@localhost;

user1@localhost can select data from columns column_name(s) and update column_name1 only the column in the table_name table.

Stored routine : GRANT EXECUTE ON PROCEDURE proc_name
TO user1@localhost;

user1@localhost can execute the stored procedure proc_name in the current database.

Proxy : GRANT PROXY ON root TO user2@localhost;

user2@localhost assumes all privileges of root.

Privileges that can be granted areselect, insert, delete, index, create, alter, drop, all, update, grant.

Revoke

To revoke some or all of the privileges which have been granted to a user before.

Syntax : REVOKE privileges ON object FROM user;

TCL

Transaction Control Language is used to manage transactions in the database and to manage the changes made to the data in a table by DML statements.

Consider a book without index page and more than 100 pages totally. You are reading the book and to take a pause you are tagging a bookmark. After sometimes you are reading from that bookmark.

So keep this scenario in the mind and consider reading the book as a transaction and pausing, tagging a bookmark are save point. Resuming the reading from the bookmark as commit. But after commit you cant rollback or cant able to read anything. It’s a lol..

TCL : Commit, Rollback, Save point

Syntax:

Start transaction ;

(update / delete / insert) query ;

savepoint savepoint_name;

(update / delete / insert) query ;

rollback to savepoint_name;

commit;

NOTE : before and after rollback, you can use select query to know the difference.

Using select query we can see that followers is 2 for content_id is 13. Now set the save point (Say s).(First table)

Update the followers as 8 for content_id is 13. See the data using select query. (Second table) Before rollback

Now, rollbacked to save point (say s) and then displayed the data present in the table using select query. (third table) After rollback

In third table, followers is 2 for content_id is 13.

Commit the transaction and now the table data shows followers is 2 for content_id is 13.

We updated the value but we rollbacked at that time followers is 2 for content_id is 13. (third table)

TitBits

SQL Designed by : Donald D. Chamberlin & Raymond F. Boyce

Appeared on : 1974.

File Extension : .sql

History

In 1970, Dr. Edgar F. “Ted” Codd of IBM described a relational model for databases and he is known as the father of relational databases.

After learning about the relational model from Edgar F. Codd, SQL was initially developed at IBM by Donald D. Chamberlin and Raymond F. Boyce.

This version is called SEQUEL (Structured English Query Language) and designed to manipulate and retrieve data stored in IBM’s System R, which was developed from Codd’s ideas.

Chamberlin and Boyce’s first attempt at a relational database language was Square. Due to subscript notation, it was difficult to use. In 1973, they began work on SEQUEL and was later changed to SQL because “SEQUEL” was a trademark of the UK-based company.

After the successful testing IBM developed commercial products based on System R.

Due to usefulness and productivity of the system Relational Software, Inc (now Oracle Corporation) developed their own SQL -based RDBMS and sells to U.S. Navy, Central Intelligence Agency, and other U.S. government agencies.

In 1979, Relational Software, Inc. introduced the first commercially available implementation of SQL, Oracle V2 (Version2) for VAX computers.

In 1986, ANSI and ISO standard groups officially adopted the SQL as standard “Database Language SQL” language definition.

Happy Learning !

From giant repos of resources to interview prep, there’s a lot of great stuff here

I have collected ten great repositories you can find on GitHub that will definitely give you an instant knowledge boost.

1. Awesome

Awesome is, without a doubt, the most popular repo that curates all topics from software development to hardware to business. It has more than 123,000 stars on Github at this moment, and one could spend days (nights) browsing it. It is my one-stop-shop if I want to learn something new.

Link to Repo

---

2. List of (Advanced) JavaScript Questions

This repository (13,000 stars) created by Lydia Hallie (great person, by the way, check her out on Instagram!) is awesome if you want to test your JavaScript knowledge and/or prepare for a job interview. Lydia constantly adds new questions, so this repo is growing steadily!

Link to Repo

---

3. JavaScript Algorithms and Data Structures

JavaScript Algorithms (59,500 stars) contains a tremendous amount of JavaScript-based examples of popular and less popular algorithms and data structures. The structure is really nice because the examples are labeled either beginner or advanced. So whether you are interested in cryptography, sorting, graphs or even machine learning (and much more), there will be something for you.

Link to Repo

---

4. Clean Code JavaScript

Knowing how to write clean code is a useful skill for almost every developer and mandatory in many projects and corporations. This repository (28,400 Stars) takes the concepts of Clean Code by Robert C. Martin and adapts them for JavaScript so you can use them in future work.

Link to Repo

---

5. free-for.dev

This great repository (30,000 stars) was created to helping developers find software (SaaS, PaaS, IaaS) and similar offerings that have free tiers. Using great software for free? Sounds awesome! There are services for email, CI/CD, monitoring, DNS, hosting, and many, many more.

Link to Repo

---

6. Tech Interview Handbook

If you want to prepare yourself for a job interview in tech (and you should!), this repository is just the right place for you. It currently has 38,000 stars on Github and helps you with:

• How to prepare for coding interviews
• Interview cheatsheet
• Algorithm tips/best practice questions
• Interview formats
• …and more

Check it out before heading to the next interview.

Link to Repo

---

7. Effective Engineer — Notes

An “Effective Engineer” is someone who gets things done. But how can you become one? This Github repository (3,700 stars) has advice for you! Topics include:

• How to optimize your learning
• How to prioritize
• Measuring your improvement

Link to Repo

---

8. List of Free Learning Resources

This repository has a whopping 135,000 stars on Github and definitely is one of the most-loved curations there. It offers lots and lots of freely available programming books, screencasts, podcasts, and even online courses of all sorts. If you are looking for learning materials — look no further!

Link to Repo

---

9. Big List of Naughty Strings

You can invest as much as you want into testing your code and programs, but when real users come into play — believe me, they will do things you would have never thought of, especially when it comes to user input. This repository (34,100 stars) has a huge list of Strings that are known to be problematic when entered as input by the user. So whether you want to impress your QA friends or uses this for testing your software, this could be quite useful!

Link to Repo

---

10. Awesome First PR Opportunities

The last repository I want to recommend to you is great if you want to get started with Open Source. Contributing to OSS is a great way to get involved with the community, add something to your portfolio, try out new things, become a better developer, and even find a new job. But getting started is quite hard, as I know from personal experience. This repository lists Open Source projects that are known for or currently have beginner-friendly issues that you can tackle. It helped me getting started with OSS, should help you as well!

I hope you find these repositories as awesome as I do, and I really hope that some of them will help you on your journey as a web developer.

You can’t pass interviews at anywhere without knowing anything about the algorithm. So why not learning now.

I am a programmer with over 15 years of experience. In my first years (from high school), I learned and worked a lot on algorithms. Until I graduated, I always studied and had a passion for it. After graduating, I started to study abroad and followed the research path.

However, after working for 3 years, I prefer to do products/software rather than research. I turned to technology/engineering instead of algorithms. However, I personally think that understanding algorithms is still something quite important for developers. And I don’t think you can pass interviews at Google or Facebook without knowing anything about them!

Coding Interview Questions | Skilled.devA full platform where I teach you everything you need to land your next job and the techniques to…skilled.dev

1. HackerRank

This site is super famous in the programming world, from junior to senior, from developer to hiring managers.

Hackerrankhas many algorithms, with difficulty from low to high, and all kinds of algorithms and data structures (even SQL). You will write code, run directly and see the results, or check the discussion to see how people solve it.

In the course of my interviews, there were also a few companies that let me come here to do their tests (algorithms + multiple choice + essay). Therefore, you can come here to do more exercises and get used to it.

2. TopCode

TopCoderis one of the first online “arenas” for programmers who can compete directly with each other. It provides a list of algorithmic challenges, which you can do online with their code editor. Single-round matches are held several times per month at a specific time, where coders compete with each other to solve challenges fastest with the highest scores.

Users who rank high on TopCoder are very good programmers and regularly participate in many programming competitions. The highest-ranking people will own their own blog called “ALGORITHMS WEEKLY by PETR MITRICHEV”, where they can write about code competitions, algorithms, math, and whatever they like.

3. Geekforgeeks

If you’ve ever had trouble doing an algorithm, after looking for a solution on Google, you’ll probably see this page a few times.

This geekforgeekspage looks a bit… ugly, the exercises are not very well classified. However, it has many solutions to common algorithm exercises. Each solution includes many different methods, together with the complexity/memory of each method. It is very interesting and helpful to read.

Therefore, sometimes you can see this page/solution example of popular algorithms home!

4. Leetcode

Another popular website for algorithm training: Leetcode. The number of problems on this is not inferior to hackerrank, there are 3 main difficulty levels are Easy/Medium/Hard (Some easy problems are very difficult).

When I went to the algorithm interview, I was often asked some Easy questions to start, Medium to evaluate the ability, and Hard is for Facebook/Google interview.

This page also has a nice one as it measures your performance/runtime code, compared to others to see if your code runs fast or slow, can be optimized or not.

Daily Coding Problem

This page I was introduced while watching Youtube Channel of CS Dojo — A Google employee who quit his job to work as a Youtuber.

Unlike the rest of the pages, this site has no assignments list. Instead, you enter your email, every day they will send you 1 algorithm to try.

These are also Easy / Medium, used by companies like Amazon / Google to interview candidates. If you purchase the Premium package, you will be given the solution.

Exercism.io

If other pages are to practice algorithms, then Exercism is a pretty good place to learn programming.

There are many exercises on Exercism, but these exercises are not too heavy on algorithms, but heavy on using programming languages:

• Use if / else, loop.
• Date processing, string processing
• Method separation, class organization
• How to handle errors, write asynchronous code

If you have programmed for a while, want to learn a new language, learn a new way of thinking, just go to Exercism!

In The End

In this article, I have shared 6 super cool websites to practice algorithms and learn programming!

I personally think that you don’t need to study too much, just access Hackerrank / Leetcode if you want to practice before the interview; Or choose Exercism if you need to learn a new language.

If you have any interesting algorithmic experience, practice interviewing, just share in the comments! Thanks.

I’ve learnd a very important thing in my whole career as a developer, IDE is my bow and extensions are my arrows.

It depends on me to choose the best bow and best arrows , of course, I choose VSCode for that and the number of best VSCode extensions.

---

There are thousands of extensions in the visual studio Marketplace, some are good and some are bad.

Let me show you the best 10 extensions and also 6 bonus extension also waiting for you down there

1. GitLens

I can say that it is the most powerful git management extensions out there. It lets you check git history, blame information, line and file history, commit searching, create or move between any branches and many other tools for handling Git.

2. Prettier

Prettier makes sure you don’t have to worry about formatting again! It’s super easy and can be configured to auto-format code on every save.

3. Live Share

It’s a real jewel when you want to collaboratively edit and debug a project or a file with others in real-time.

You can share your code, terminal and server with other users and if you use the live audio extension then you can even add… audio to your communication.

4. Debugger for Chrome

Good debugging is more than endless console logs. Chrome offers an analog to its chrome extension for VSCode.

5. ESLint

If you need linter and a formatter in one pack than this is for you.

It can auto-format your code and find the bugs in your code as well.

6. Visual Studio IntelliCode

It is built for helping developers and programmers with smart code completion suggestions.

it comes with default support for Python, TypeScript/JavaScript, React, and Java.

7. SQLTools — Database tools

A tool for managing DB via VSCode.

It supports a number of drivers and you can do many things by using it like Connection explorer, query runner, intellisense, bookmarks, query history.

8. Remote — SSH

This is one of the best tools out there. You can open any folder on the remote machine and work with it just as you would if the folder was on your own machine.

No source code needs to be on your local machine to gain these benefits since the extension runs commands and other extensions directly on the remote machine

9. Settings Sync

It will come very handy to you when you switch to a new vscode environment instead of configuring everything from start, you can sync them between your every vscode environments.

10. Live Server

When writing code, we often go back and forth between our IDE and the browser, to examine the result of our latest code updates.

Live Server makes sure your browser refreshes when it needs to.

---

Bonus 1. Bracket Pair Colorizer 2

This extension colors pairs of brackets in different colors to make it easy to identify where each couple of brackets open and end.

2. Better Comments

The Better Comments extension will help you create more human-friendly comments in your code.
Each color can serve as a means to signify the type of comment (caution, to-do’s, etc.)

3. CodeStream

One of my favorites. A great way to discuss code.

4. Code Spell Checker

A great tool to validate our comments spelling.

5.Polacode

A great way to screenshot your code. Very useful when writing tutorials.

6. Path Intellisense

In large projects, we can’t remember every file path. That’s where path intellisense comes into play.

After installing this we have to just type the path it will show us the next directories and file we can use.

Bonus: Bit.dev

Bit.dev is not a VSCode extension but a very powerful tool every frontend developer should know.

Bit is a cloud component hub. It is where you publish, document and organize all your reusable components.

With Bit, there’s no need to create a whole component library project — simply publish individual components from whatever codebase you’re working on. Use it with your team to maximize code reuse, speed up development and quite simply, build a more scalable app.

Bit.dev supports React, Vue, Angular, React Native, React with TS, and many others.

A quick introduction to 8 commonly used data structures

Data Structures are a specialized means of organizing and storing data in computers in such a way that we can perform operations on the stored data more efficiently. Data structures have a wide and diverse scope of usage across the fields of Computer Science and Software Engineering.

Data structures are being used in almost every program or software system that has been developed. Moreover, data structures come under the fundamentals of Computer Science and Software Engineering. It is a key topic when it comes to Software Engineering interview questions. Hence as developers, we must have good knowledge about data structures.

In this article, I will be briefly explaining 8 commonly used data structures every programmer must know.

1. Arrays

An array is a structure of fixed-size, which can hold items of the same data type. It can be an array of integers, an array of floating-point numbers, an array of strings or even an array of arrays (such as 2-dimensional arrays). Arrays are indexed, meaning that random access is possible.

Array operations

• Traverse: Go through the elements and print them.
• Search: Search for an element in the array. You can search the element by its value or its index
• Update: Update the value of an existing element at a given index

Inserting elements to an array and deleting elements from an array cannot be done straight away as arrays are fixed in size. If you want to insert an element to an array, first you will have to create a new array with increased size (current size + 1), copy the existing elements and add the new element. The same goes for the deletion with a new array of reduced size.

Applications of arrays

• Used as the building blocks to build other data structures such as array lists, heaps, hash tables, vectors and matrices.
• Used for different sorting algorithms such as insertion sort, quick sort, bubble sort and merge sort.

2. Linked Lists

A linked list is a sequential structure that consists of a sequence of items in linear order which are linked to each other. Hence, you have to access data sequentially and random access is not possible. Linked lists provide a simple and flexible representation of dynamic sets.

Let’s consider the following terms regarding linked lists. You can get a clear idea by referring to Figure 2.

• Elements in a linked list are known as nodes.
• Each node contains a key and a pointer to its successor node, known as next.
• The attribute named head points to the first element of the linked list.
• The last element of the linked list is known as the tail.

Following are the various types of linked lists available.

• Singly linked list — Traversal of items can be done in the forward direction only.
• Doubly linked list — Traversal of items can be done in both forward and backward directions. Nodes consist of an additional pointer known as prev, pointing to the previous node.
• Circular linked lists — Linked lists where the prev pointer of the head points to the tail and the next pointer of the tail points to the head.

Linked list operations

• Search: Find the first element with the key k in the given linked list by a simple linear search and returns a pointer to this element
• Insert: Insert a key to the linked list. An insertion can be done in 3 different ways; insert at the beginning of the list, insert at the end of the list and insert in the middle of the list.
• Delete: Removes an element x from a given linked list. You cannot delete a node by a single step. A deletion can be done in 3 different ways; delete from the beginning of the list, delete from the end of the list and delete from the middle of the list.

Applications of linked lists

• Used for symbol table management in compiler design.
• Used in switching between programs using Alt + Tab (implemented using Circular Linked List).

3. Stacks

A stack is a LIFO (Last In First Out — the element placed at last can be accessed at first) structure which can be commonly found in many programming languages. This structure is named as “stack” because it resembles a real-world stack — a stack of plates.

Stack operations

Given below are the 2 basic operations that can be performed on a stack. Please refer to Figure 3 to get a better understanding of the stack operations.

• Push: Insert an element on to the top of the stack.
• Pop: Delete the topmost element and return it.

Furthermore, the following additional functions are provided for a stack in order to check its status.

• Peek: Return the top element of the stack without deleting it.
• isEmpty: Check if the stack is empty.
• isFull: Check if the stack is full.

Applications of stacks

• Used for expression evaluation (e.g.: shunting-yard algorithm for parsing and evaluating mathematical expressions).
• Used to implement function calls in recursion programming.

4. Queues

A queue is a FIFO (First In First Out — the element placed at first can be accessed at first) structure which can be commonly found in many programming languages. This structure is named as “queue” because it resembles a real-world queue — people waiting in a queue.

Queue operations

Given below are the 2 basic operations that can be performed on a queue. Please refer to Figure 4 to get a better understanding of the queue operations.

• Enqueue: Insert an element to the end of the queue.
• Dequeue: Delete the element from the beginning of the queue.

Applications of queues

• Used to manage threads in multithreading.
• Used to implement queuing systems (e.g.: priority queues).

5. Hash Tables

A Hash Table is a data structure that stores values which have keys associated with each of them. Furthermore, it supports lookup efficiently if we know the key associated with the value. Hence it is very efficient in inserting and searching, irrespective of the size of the data.

Direct Addressing uses the one-to-one mapping between the values and keys when storing in a table. However, there is a problem with this approach when there is a large number of key-value pairs. The table will be huge with so many records and may be impractical or even impossible to be stored, given the memory available on a typical computer. To avoid this issue we use hash tables.

Hash Function

A special function named as the hash function (h) is used to overcome the aforementioned problem in direct addressing.

In direct accessing, a value with key k is stored in the slot k. Using the hash function, we calculate the index of the table (slot) to which each value goes. The value calculated using the hash function for a given key is called the hash value which indicates the index of the table to which the value is mapped.

h(k) = k % m

• h: Hash function
• k: Key of which the hash value should be determined
• m: Size of the hash table (number of slots available). A prime value that is not close to an exact power of 2 is a good choice for m.

Consider the hash function h(k) = k % 20, where the size of the hash table is 20. Given a set of keys, we want to calculate the hash value of each to determine the index where it should go in the hash table. Consider we have the following keys, the hash and the hash table index.

• 1 → 1%20 → 1
• 5 → 5%20 → 5
• 23 → 23%20 → 3
• 63 → 63%20 → 3

From the last two examples given above, we can see that collision can arise when the hash function generates the same index for more than one key. We can resolve collisions by selecting a suitable hash function h and use techniques such as chaining and open addressing.

Applications of hash tables

• Used to implement database indexes.
• Used to implement associative arrays.
• Used to implement the “set” data structure.

6. Trees

A tree is a hierarchical structure where data is organized hierarchically and are linked together. This structure is different than a linked list whereas, in a linked list, items are linked in a linear order.

Various types of trees have been developed throughout the past decades, in order to suit certain applications and meet certain constraints. Some examples are binary search tree, B tree, treap, red-black tree, splay tree, AVL tree and n-ary tree.

Binary Search Trees

A binary search tree (BST), as the name suggests, is a binary tree where data is organized in a hierarchical structure. This data structure stores values in sorted order.

Every node in a binary search tree comprises the following attributes.

1. key: The value stored in the node.
2. left: The pointer to the left child.
3. right: The pointer to the right child.
4. p: The pointer to the parent node.

A binary search tree exhibits a unique property that distinguishes it from other trees. This property is known as the binary-search-tree property.

Let x be a node in a binary search tree.

• If y is a node in the left subtree of x, then y.key ≤ x.key
• If y is a node in the right subtree of x, then y.key ≥ x.key

Applications of trees

• Binary Trees: Used to implement expression parsers and expression solvers.
• Binary Search Tree: used in many search applications where data are constantly entering and leaving.
• Heaps: used by JVM (Java Virtual Machine) to store Java objects.
• Treaps: used in wireless networking.

Check my articles below on 8 useful tree data structures and self-balancing binary search trees.8 Useful Tree Data Structures Worth KnowingAn overview of 8 different tree data structurestowardsdatascience.comSelf-Balancing Binary Search Trees 101Introduction to Self-Balancing Binary Search Treestowardsdatascience.com

7. Heaps

A Heap is a special case of a binary tree where the parent nodes are compared to their children with their values and are arranged accordingly.

Let us see how we can represent heaps. Heaps can be represented using trees as well as arrays. Figures 7 and 8 show how we can represent a binary heap using a binary tree and an array.

Heaps can be of 2 types.

1. Min Heap — the key of the parent is less than or equal to those of its children. This is called the min-heap property. The root will contain the minimum value of the heap.
2. Max Heap — the key of the parent is greater than or equal to those of its children. This is called the max-heap property. The root will contain the maximum value of the heap.

Applications of heaps

• Used in heapsort algorithm.
• Used to implement priority queues as the priority values can be ordered according to the heap property where the heap can be implemented using an array.
• Queue functions can be implemented using heaps within O(log n) time.
• Used to find the kᵗʰ smallest (or largest) value in a given array.

Check my article below on implementing a heap using the python heapq module.Introduction to Python Heapq ModuleA simple introduction on how to use Python’s heapq moduletowardsdatascience.com

8. Graphs

A graph consists of a finite set of vertices or nodes and a set of edges connecting these vertices.

The order of a graph is the number of vertices in the graph. The size of a graph is the number of edges in the graph.

Two nodes are said to be adjacent if they are connected to each other by the same edge.

Directed Graphs

A graph G is said to be a directed graph if all its edges have a direction indicating what is the start vertex and what is the end vertex.

We say that (u, v) is incident from or leaves vertex u and is incident to or enters vertex v.

Self-loops: Edges from a vertex to itself.

Undirected Graphs

A graph G is said to be an undirected graph if all its edges have no direction. It can go in both ways between the two vertices.

If a vertex is not connected to any other node in the graph, it is said to be isolated.

Applications of graphs

• Used to represent social media networks. Each user is a vertex, and when users connect they create an edge.
• Used to represent web pages and links by search engines. Web pages on the internet are linked to each other by hyperlinks. Each page is a vertex and the hyperlink between two pages is an edge. Used for Page Ranking in Google.
• Used to represent locations and routes in GPS. Locations are vertices and the routes connecting locations are edges. Used to calculate the shortest route between two locations.

Final Thoughts

A cheat sheet for the time complexities of the data structure operations can be found in this link. Moreover, check out my article below where I have implemented a few common data structures from scratch using C++.Data Structures in C++ — Part 1Implementing common data structures in C++towardsdatascience.com

I hope you found this article useful as a simple introduction to data structures. I would love to hear your thoughts.

Thanks a lot for reading.

Cheers!

References

[1] Introduction to Algorithms, Third Edition By Thomas H. Cormen, Charles E. Leiserson, Ronald L. Rivest and Clifford Stein.

[2] List of data structures from Wikipedia (https://en.wikipedia.org/wiki/List_of_data_structures)

Whether you’ve come across these terms online or have heard it thrown around by your web designer, it is important to know the difference so that you can determine the best solution for your website.

In simplest terms, static Website pages are those with content that cannot change without a developer editing its source code, while dynamic Website pages can display different content from the same source code. When it comes to using static or dynamic pages for parts of your company’s website, having the most advanced code on each of your pages is not important. What is important is the purpose each page serves for your website.

Static Websites

Static websites contain fixed number of pages and format of web page is fixed which delivers information to the client. There are 110 changes in contents of web page while page is running on client’s browser. This kind of web sites created from HTML and CSS coding on simple text editor like notepad. Example an organization site, institute site etc.

Static Website pages display the exact same information whenever anyone visits it. Static Website pages do not have to be simple plain text. They can feature detailed multimedia design and even videos. However, every visitor to that page will be greeted by the exact same text, multimedia design or video every time he visits the page until you alter that page’s source code.

Dynamic Website Pages

Dynamic websites can change the web page contents dynamically while the page is running on client’s browser. This kind of websites use server- side programming like PHP, Asp.NET. and JSP etc. to modify page contents on run time. Dynamic websites use client side scripting for prepare dynamic design and server- side code to handle event, manage session and cookies, and storing and retrieving data from database. Example E-commerce sites, online form application, E-governance site, social networking sites etc.

Dynamic Website pages are capable of producing different content for different visitors from the same source code file. The website can display different content based on what operating system or browser the visitor is using, whether she is using a PC or a mobile device, or even the source that referred the visitor. A dynamic Website page is not necessarily better than a static Website page. The two simply serve different purposes.

Dynamic Page Use

Dynamic pages can serve a variety of purposes. For example, websites run by content management systems allow a single source code file to load the content of many different possible pages. Content creators use a gateway page to submit the material for new pages into the CMS’ database. The dynamic page can then load the material for any page in the database, based on parameters in the URL with which a visitor requests the page. Dynamic pages are also what let users log into websites to see personalized content.

Static vs Dynamic Creation

Developers generally create static pages with HTML, but use languages like PHP, Javascript, or Actionscript to create dynamic pages. They can also use frameworks like Ruby on Rails, PHP or Flex for dynamic pages. Dynamic languages and frameworks also have the technical capacity to create static Web page content. However, doing so creates source code that is unnecessarily complex for its purpose while being more difficult to maintain.

• In static web pages the theme and content of web pages remained fixed and in dynamic web pages they changed according to run time.
• Browsing and loading of static web pages are more quicker than dynamic web pages because unlike dynamic web pages they do not required the request of server.
• Changing the content in static web pages is a difficult task because you have to develop and upload a new page while in dynamic web pages server application do this automatically.
• If file extension of a URL is in .htm or .html then it is a static web pages. While if it is in .php, .asp and .jsp then it is the example of dynamic web pages.
• Static web pages are created through HTML language while dynamic web pages are created by the usage of PHP, JavaScript and Actionscript languages.
• Static web pages plan is an easy and cheap method if you wants to create a static and non updating web pages. While dynamic web pages method is advisable if you have plan to update the content and material frequently.

Which Should You Choose?

Many people prefer dynamic websites because they have a lot of benefits. Dynamic sites reduce ongoing maintenance costs, make data management very efficient, and enable the addition of any future add-ons such as data feeds or a comprehensive site search. They also make it impossible to destroy the layout, as might happen if you edit it from a web page editor.

There are some drawbacks to using dynamic websites. First, they usually cost more to develop, because they require more complex coding, as well as the development of a content management interface to enable you to maintain your website. Second, you will need to obtain web hosting which supports databases and dynamic languages. Fortunately, most hosts do offer these features by default.

A quick introduction to 8 commonly used data structures

Data Structures are a specialized means of organizing and storing data in computers in such a way that we can perform operations on the stored data more efficiently. Data structures have a wide and diverse scope of usage across the fields of Computer Science and Software Engineering.

Data structures are being used in almost every program or software system that has been developed. Moreover, data structures come under the fundamentals of Computer Science and Software Engineering. It is a key topic when it comes to Software Engineering interview questions. Hence as developers, we must have good knowledge about data structures.

In this article, I will be briefly explaining 8 commonly used data structures every programmer must know.

1. Arrays

An array is a structure of fixed-size, which can hold items of the same data type. It can be an array of integers, an array of floating-point numbers, an array of strings or even an array of arrays (such as 2-dimensional arrays). Arrays are indexed, meaning that random access is possible.

Array operations

• Traverse: Go through the elements and print them.
• Search: Search for an element in the array. You can search the element by its value or its index
• Update: Update the value of an existing element at a given index

Inserting elements to an array and deleting elements from an array cannot be done straight away as arrays are fixed in size. If you want to insert an element to an array, first you will have to create a new array with increased size (current size + 1), copy the existing elements and add the new element. The same goes for the deletion with a new array of reduced size.

Applications of arrays

• Used as the building blocks to build other data structures such as array lists, heaps, hash tables, vectors and matrices.
• Used for different sorting algorithms such as insertion sort, quick sort, bubble sort and merge sort.

2. Linked Lists

A linked list is a sequential structure that consists of a sequence of items in linear order which are linked to each other. Hence, you have to access data sequentially and random access is not possible. Linked lists provide a simple and flexible representation of dynamic sets.

Let’s consider the following terms regarding linked lists. You can get a clear idea by referring to Figure 2.

• Elements in a linked list are known as nodes.
• Each node contains a key and a pointer to its successor node, known as next.
• The attribute named head points to the first element of the linked list.
• The last element of the linked list is known as the tail.

Following are the various types of linked lists available.

• Singly linked list — Traversal of items can be done in the forward direction only.
• Doubly linked list — Traversal of items can be done in both forward and backward directions. Nodes consist of an additional pointer known as prev, pointing to the previous node.
• Circular linked lists — Linked lists where the prev pointer of the head points to the tail and the next pointer of the tail points to the head.

Linked list operations

• Search: Find the first element with the key k in the given linked list by a simple linear search and returns a pointer to this element
• Insert: Insert a key to the linked list. An insertion can be done in 3 different ways; insert at the beginning of the list, insert at the end of the list and insert in the middle of the list.
• Delete: Removes an element x from a given linked list. You cannot delete a node by a single step. A deletion can be done in 3 different ways; delete from the beginning of the list, delete from the end of the list and delete from the middle of the list.

Applications of linked lists

• Used for symbol table management in compiler design.
• Used in switching between programs using Alt + Tab (implemented using Circular Linked List).

3. Stacks

A stack is a LIFO (Last In First Out — the element placed at last can be accessed at first) structure which can be commonly found in many programming languages. This structure is named as “stack” because it resembles a real-world stack — a stack of plates.

Stack operations

Given below are the 2 basic operations that can be performed on a stack. Please refer to Figure 3 to get a better understanding of the stack operations.

• Push: Insert an element on to the top of the stack.
• Pop: Delete the topmost element and return it.

Furthermore, the following additional functions are provided for a stack in order to check its status.

• Peek: Return the top element of the stack without deleting it.
• isEmpty: Check if the stack is empty.
• isFull: Check if the stack is full.

Applications of stacks

• Used for expression evaluation (e.g.: shunting-yard algorithm for parsing and evaluating mathematical expressions).
• Used to implement function calls in recursion programming.

4. Queues

A queue is a FIFO (First In First Out — the element placed at first can be accessed at first) structure which can be commonly found in many programming languages. This structure is named as “queue” because it resembles a real-world queue — people waiting in a queue.

Queue operations

Given below are the 2 basic operations that can be performed on a queue. Please refer to Figure 4 to get a better understanding of the queue operations.

• Enqueue: Insert an element to the end of the queue.
• Dequeue: Delete the element from the beginning of the queue.

Applications of queues

• Used to manage threads in multithreading.
• Used to implement queuing systems (e.g.: priority queues).

5. Hash Tables

A Hash Table is a data structure that stores values which have keys associated with each of them. Furthermore, it supports lookup efficiently if we know the key associated with the value. Hence it is very efficient in inserting and searching, irrespective of the size of the data.

Direct Addressing uses the one-to-one mapping between the values and keys when storing in a table. However, there is a problem with this approach when there is a large number of key-value pairs. The table will be huge with so many records and may be impractical or even impossible to be stored, given the memory available on a typical computer. To avoid this issue we use hash tables.

Hash Function

A special function named as the hash function (h) is used to overcome the aforementioned problem in direct addressing.

In direct accessing, a value with key k is stored in the slot k. Using the hash function, we calculate the index of the table (slot) to which each value goes. The value calculated using the hash function for a given key is called the hash value which indicates the index of the table to which the value is mapped.

h(k) = k % m

• h: Hash function
• k: Key of which the hash value should be determined
• m: Size of the hash table (number of slots available). A prime value that is not close to an exact power of 2 is a good choice for m.

Consider the hash function h(k) = k % 20, where the size of the hash table is 20. Given a set of keys, we want to calculate the hash value of each to determine the index where it should go in the hash table. Consider we have the following keys, the hash and the hash table index.

• 1 → 1%20 → 1
• 5 → 5%20 → 5
• 23 → 23%20 → 3
• 63 → 63%20 → 3

From the last two examples given above, we can see that collision can arise when the hash function generates the same index for more than one key. We can resolve collisions by selecting a suitable hash function h and use techniques such as chaining and open addressing.

Applications of hash tables

• Used to implement database indexes.
• Used to implement associative arrays.
• Used to implement the “set” data structure.

6. Trees

A tree is a hierarchical structure where data is organized hierarchically and are linked together. This structure is different than a linked list whereas, in a linked list, items are linked in a linear order.

Various types of trees have been developed throughout the past decades, in order to suit certain applications and meet certain constraints. Some examples are binary search tree, B tree, treap, red-black tree, splay tree, AVL tree and n-ary tree.

Binary Search Trees

A binary search tree (BST), as the name suggests, is a binary tree where data is organized in a hierarchical structure. This data structure stores values in sorted order.

Every node in a binary search tree comprises the following attributes.

1. key: The value stored in the node.
2. left: The pointer to the left child.
3. right: The pointer to the right child.
4. p: The pointer to the parent node.

A binary search tree exhibits a unique property that distinguishes it from other trees. This property is known as the binary-search-tree property.

Let x be a node in a binary search tree.

• If y is a node in the left subtree of x, then y.key ≤ x.key
• If y is a node in the right subtree of x, then y.key ≥ x.key

Applications of trees

• Binary Trees: Used to implement expression parsers and expression solvers.
• Binary Search Tree: used in many search applications where data are constantly entering and leaving.
• Heaps: used by JVM (Java Virtual Machine) to store Java objects.
• Treaps: used in wireless networking.

Check my articles below on 8 useful tree data structures and self-balancing binary search trees.8 Useful Tree Data Structures Worth KnowingAn overview of 8 different tree data structurestowardsdatascience.comSelf-Balancing Binary Search Trees 101Introduction to Self-Balancing Binary Search Treestowardsdatascience.com

7. Heaps

A Heap is a special case of a binary tree where the parent nodes are compared to their children with their values and are arranged accordingly.

Let us see how we can represent heaps. Heaps can be represented using trees as well as arrays. Figures 7 and 8 show how we can represent a binary heap using a binary tree and an array.

Heaps can be of 2 types.

1. Min Heap — the key of the parent is less than or equal to those of its children. This is called the min-heap property. The root will contain the minimum value of the heap.
2. Max Heap — the key of the parent is greater than or equal to those of its children. This is called the max-heap property. The root will contain the maximum value of the heap.

Applications of heaps

• Used in heapsort algorithm.
• Used to implement priority queues as the priority values can be ordered according to the heap property where the heap can be implemented using an array.
• Queue functions can be implemented using heaps within O(log n) time.
• Used to find the kᵗʰ smallest (or largest) value in a given array.

Check my article below on implementing a heap using the python heapq module.Introduction to Python Heapq ModuleA simple introduction on how to use Python’s heapq moduletowardsdatascience.com

8. Graphs

A graph consists of a finite set of vertices or nodes and a set of edges connecting these vertices.

The order of a graph is the number of vertices in the graph. The size of a graph is the number of edges in the graph.

Two nodes are said to be adjacent if they are connected to each other by the same edge.

Directed Graphs

A graph G is said to be a directed graph if all its edges have a direction indicating what is the start vertex and what is the end vertex.

We say that (u, v) is incident from or leaves vertex u and is incident to or enters vertex v.

Self-loops: Edges from a vertex to itself.

Undirected Graphs

A graph G is said to be an undirected graph if all its edges have no direction. It can go in both ways between the two vertices.

If a vertex is not connected to any other node in the graph, it is said to be isolated.

Applications of graphs

• Used to represent social media networks. Each user is a vertex, and when users connect they create an edge.
• Used to represent web pages and links by search engines. Web pages on the internet are linked to each other by hyperlinks. Each page is a vertex and the hyperlink between two pages is an edge. Used for Page Ranking in Google.
• Used to represent locations and routes in GPS. Locations are vertices and the routes connecting locations are edges. Used to calculate the shortest route between two locations.

Final Thoughts

A cheat sheet for the time complexities of the data structure operations can be found in this link. Moreover, check out my article below where I have implemented a few common data structures from scratch using C++.Data Structures in C++ — Part 1Implementing common data structures in C++towardsdatascience.com

I hope you found this article useful as a simple introduction to data structures. I would love to hear your thoughts.

Thanks a lot for reading.

Cheers!

References

[1] Introduction to Algorithms, Third Edition By Thomas H. Cormen, Charles E. Leiserson, Ronald L. Rivest and Clifford Stein.

[2] List of data structures from Wikipedia (https://en.wikipedia.org/wiki/List_of_data_structures)

Digital transformation will create trillions of dollars of value. While estimates vary, the World Economic Forum in 2016 estimated an increase in $100 trillion in global business and social value by 2030. Due to AI, PwC has estimated an increase of $15.7 trillion and McKinsey has estimated an increase of $13 trillion in annual global GDP by 2030. We are currently in the middle of an AI renaissance, driven by big data and breakthroughs in machine learning and deep learning. These breakthroughs offer opportunities and challenges to companies depending on the speed at which they adapt to these changes.

Modern enterprises face 5 key challenges in today’s era of big data

1. Handling a multiplicity of enterprise source systems

The average Fortune 500 enterprise has a few hundred enterprise IT systems, all with their different data formats, mismatched references across data sources, and duplication

2. Incorporating and contextualising high frequency data

The challenge gets significantly harder with increase in sensoring, resulting inflows of real time data. For example, readings of the gas exhaust temperature for an offshore low-pressure compressor are only of limited value in of itself. But combined with ambient temperature, wind speed, compressor pump speed, history of previous maintenance actions, and maintenance logs, this real-time data can create a valuable alarm system for offshore rig operators.

3. Working with data lakes

Today, storing large amounts of disparate data by putting it all in one infrastructure location does not reduce data complexity any more than letting data sit in siloed enterprise systems. 

4. Ensuring data consistency, referential integrity, and continuous downstream use

A fourth big data challenge is representing all existing data as a unified image, keeping this image updated in real-time and updating all downstream analytics that use these data. Data arrival rates vary by system, data formats from source systems change, and data arrive out of order due to networking delays.

5. Enabling new tools and skills for new needs

Enterprise IT and analytics teams need to provide tools that enable employees with different levels of data science proficiency to work with large data sets and perform predictive analytics using a unified data image.

Let’s look at what’s involved in developing and deploying AI applications at scale

Data assembly and preparation

The first step is to identify the required and relevant data sets and assemble them. There are often issues with data duplication, gaps in data, unavailable data and data out of sequence.

Feature engineering

This involves going through the data and crafting individual signals that the data scientists and domain experts think will be relevant to the problem being solved. In the case of AI-based predictive maintenance, signals could include the count of specific fault alarms over the trailing 7 days,14 days and 21 days, the sum of the specific alarms over the same trailing periods; and the maximum value of certain sensor signals over those trailing periods. 

Labelling the outcomes

This step involves labeling the outcomes the model tries to predict. For example, in AI-based predictive maintenance applications, source data sets rarely identify actual failure labels, and practitioners have to infer failure points based on a  combination of factors such as fault codes and technician work orders.

Setting up the training data

For classification tasks, data scientists need to ensure that labels are appropriately balanced with positive and negative examples to provide the classifier algorithm enough balanced data. Data scientists also need to ensure the classifier is not biased with artificial patterns in the data.

Choosing and training the algorithm

Numerous algorithm libraries are available to data scientists today, created by companies, universities, research organizations, government agencies and individual contributors.

Deploying the algorithm into production

Machine learning algorithms, once deployed, need to receive new data, generate outputs, and have some actions or decisions be made based on those outputs. This may mean embedding the algorithm within an enterprise application used by humans to make decisions – for example, a predictive maintenance application that identifies and prioritizes equipment requiring maintenance to provide guidance for maintenance crews. This is where the real value is created – by reducing equipment downtime and servicing costs through more accurate failure prediction that enables proactive maintenance before the equipment actually fails. In order for the machine learning algorithms to operate in production, the underlying compute infrastructure needs to be set up and managed. 

Close-loop continuous improvement

Algorithms typically require frequent retraining by data science teams. As market conditions change, business objects and processes evolve, and new data sources are identified. Organizations need to rapidly develop, retrain, and deploy new models as circumstances change.

Therefore, problems that have to be addressed to solve AI computing problems are nontrivial. Massively parallel elastic computing and storage capacity are prerequisites. In addition to the cloud, there is a multiplicity of data services necessary to develop, provision, and operate applications of this nature. However, the price of missing a transformational strategic shift is steep. The corporate graveyard is littered with once-great companies that failed to change.

This article originally appeared on Makeen Technologies.

From last few days, I have been writing on Microservices Architecture and Container Technology. You can read that here on Part-One & Part-Two. During this, I have come across many docker terminologies and having a quick understanding of these terminology helps a lot.

1. Docker Editions

・Docker Community Edition (CE) is ideal for Developers who are looking for experimenting docker and creating container-based applications. It’s free.

・Docker Enterprise Edition (EE) is a Containers-as-a-Service (CaaS) platform. Enterprise Edition Subscription packages include an integrated Docker platform and tooling for container management and security. For Pricing refer — https://www.docker.com/pricing

2. Docker ID

Your free Docker ID grants you access to Docker services such as the Docker Store, Docker Cloud, Docker Hub repositories, and some beta programs. Your Docker ID becomes repository namespace used by hosted services such as Docker Hub and Docker Cloud. All you need is an email address. This account also allows you to log in to services such as the Docker Support Center, the Docker Forums, and the Docker Success portal.

3. Docker Hub

Docker Hub (Like GitHub) is a cloud-based registry service which allows you to link to code repositories, build your images and test them, stores manually pushed images, and links to Docker Cloud so you can deploy images to your hosts. It provides a centralized resource for container image discovery, distribution and change management, user and team collaboration, and workflow automation throughout the development pipeline.

4. Docker Store

For publishers, Docker Store is the best way for you to distribute and sell your Dockerized content.

5. Docker Cloud

Docker Cloud uses the hosted Docker Cloud Registry, which allows you to publish Dockerized images on the internet either publicly or privately. Docker Cloud can also store pre-built images, or link to your source code so it can build the code into Docker images, and optionally test the resulting images before pushing them to a repository. Docker cloud has more features then Docker Hub.

Docker Cloud uses Docker Hub as its native registry for storing both public and private repositories. Once you push your images to Docker Hub, they will be available in Docker Cloud.

6. Docker Image & Container

・Everything starts with the Dockerfile (source code of the image).
・You build Dockerfile to create the Image.
・Redistribution is done using the Registry (ex: https://hub.docker.com).
・Running Image is Container. If Image is a class then the container is an instance.
Dockerfile > (Build) > Image > Registry(optional) > (Run) > Container.

7. Docker Engine

When people say “Docker” they typically mean Docker Engine. Docker Engine accepts docker commands from the CLI, such asdocker run <image>, docker ps to list running containers, docker images to list images, and so on.

Docker Engine is a client-server application with these major components:
A server which is a type of long-running program called a daemon process (the dockerd command).
A REST API which specifies interfaces that programs can use to talk to the daemon and instruct it what to do.
A command line interface (CLI) client (the docker command).

8. Docker Machine

Docker Machine is a tool that lets you install Docker Engine on virtual hosts, and manage the hosts with docker-machinecommands. You can use Machine to create Docker hosts on your local Mac or Windows box, on your company network, in your data center, or on cloud providers like Azure, AWS, or Digital Ocean.

Docker Machine has these two broad use cases.

9. Docker Compose

Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a YAML file to configure your application’s services. Then, with a single command, you create and start all the services from your configuration.

Thanks for reading

Please do follow and subscribe to my page.