Page 1 of 74

What is ExpressJS → What is Express

• Using Middleware
• Working wit → Working with Requests & Responses
• Routing
• Returning HTML Pages (Files)

---

Express FRAMEWORK

• Server Logic is Complex:
  • Data Buffer
  • Different Headers
  • Headers & HTML

---

Express heavily depends upon Middleware:

• Request: Incoming Request is funneled through a bunch of functions through Express.js.

• Middleware:

  next()

  app.use()

  Method for defining middleware.

  • Middleware to be send → to be sent
  • Response

---

Types of Middleware

1. app.use (Application level)
2. router.use (Router level)
3. express.json, express.urlencoded (Built-in)
4. bodyParser, cookieParser (3rd Party middleware)

---

• res.sendFile() takes absolute path.

---

Page 2 of 74

Serving Static Files

• Make an exception for a folder to serve static files (access file system):

  app.use(express.static(path.join(__dirname, 'public')))

---

Managing Data

• Render Dynamic Content in our Views

---

Templating Engines

• Understanding Templating Engines

Templating Engines

• HTML-ish Template

  flowchart TD
    A[Node/Express Content] --> B[Templating Engine]
    B --> C[Replaces Placeholders/Scripts with HTML Content]
    C --> D[HTML File]

• e.g., EJS, Pug (Jade), Handlebars

  app.set('view engine', 'ejs')
  app.set('views', <folder>)
  res.render(<file>, <data>)

---

Page 3 of 74

MVC Architecture

---

flowchart TD
    A[Request] --> B[Routes]
    B --> C[Browser Page]
    B --> D[Model] 
    D --> E[Database]
    D --> F[Controller]
    F --> G[View]

---

• Model: It can be anything related to data
  • File
  • Simple Class
  • SQL
  • Mongo

---

Dynamic Routes

• /:id → Anything, and now name is id (req.params.id)

---

Page 4 of 74

Databases

• Create SQL, parse & make it easily accessible
  • Use a Database Or → Or store from/with a file

---

---

What's SQL? (Structured Query Language)

• Data is stored in tables
• Each table has columns/attributes
• Each data is represented as row/field (seconds → record)

---

Characteristics

• Strong Data Schema: id, name, age
• Data Relations: If connected tables
  • One to One
  • One to Many
  • Many to Many

---

What's NoSQL?

• Data is stored in collections
• Each collection has documents
• Documents look like JavaScript objects

Characteristics

• Weak Data Relation
• No strong data schema

---

Page 5 of 74

Horizontal Scaling vs Vertical Scaling

• Horizontal scaling: Add more servers (can manage more data in one DB)
• Vertical scaling: Improve server capacity (hardware)

---

---

MongoDB

erDiagram
    database ||--o{ collection : contains
    collection {
        string users
        string orders
    }
    users {
        string name
        int age
        string (schema?)
    }
    orders {
        ...
    }

• Schema-less

---

Page 6 of 74

JSON Format

• JSON format looks like BSON format actually (Binary JSON)

  {
    "key": "value",
    "arr": [
      { /* object */ },
      { /* object */ }
    ]
  }

---

MongoDB fetches other relations, not actually duplicating them nor do they relate them.
They just embed it and whenever we need it, it fetches to make it fast & efficient.

---

Hence, Relations in MongoDB can be called as:

---

Page 7 of 74

Shell

$ sudo service mongod start
$ mongo
# show dbs
# use <new db>

& many more commands.

---

Drivers

Drivers are just the ecosystem allowing different languages to use with the MongoDB server.

---

Working with MongoDB

flowchart LR
    Application -->|Request| BackendServer -->|Queries| MongoDBServer
    Devices -->|Queries| MongoDBServer
    MongoDBServer --> StorageEngine -->|File/Data access| DBFiles
    MongoDBShell -->|playground & administration| MongoDBServer

---

Page 8 of 74

CRUD: Create, Read, Update, Delete

Create

insertOne(data, options)
insertMany(data, options)

Read

find(filter, options)
findOne(filter, options)

---

Update

updateOne(filter, data, options)
updateMany(filter, data, options)
replaceOne(filter, data, options)

---

Delete

deleteOne(filter, options)
deleteMany(filter, options)

---

Basic Commands

show dbs
show collections
use <db-name>
db.dropDatabase()

db.<collection-name>.insertOne(<object>)
db.<collection-name>.insertMany([array of objects])
db.<collection-name>.find({ filter }).pretty()

• Filter can be anything

{ "distance": 100 }
{ "distance": { $gt: 100 } } // greater than 100

• .findOne(filter)

---

Page 9 of 74

Update Operations

db.<collection-name>.updateOne(filter, { atomic operator })
  // $set: { option: "value" }
db.<collection-name>.updateMany(filter, { ... })
db.<collection-name>.update(filter, object) // changes the whole object
db.<collection-name>.replaceOne(filter, object) // like update but more aggressive

• Delete is same as insert or read.

---

• find() gives a cursor which can be used to apply other methods:

  toArray()
  forEach((c) => { ... })

---

PROJECTION

db.<collection-name>.find({ f1, f2 }, { key: 1 }).limit(5)

• Keys: 0 (not included)
• Cursor modifiers

---

Page 10 of 74

EMBEDDED DOCUMENTS

Data

• Array

  {
      "hobbies": ["sports", "cooking"]
  }

Access

• findOne({}, {}).hobbies
• find({ hobbies: "sports" }) (lookup in array)

---

• Array of Objects

  {
      "hobbies": [
          { "sports": ["cricket", "badminton"] },
          { "description": "...", "status": "lazy" }
      ]
  }

---

```json
{
    "hobbies": [
        { "sports": ["cricket", "swim", "champ"] }
    ]
}
```

• find({ "hobbies.sports": "cricket" })

---

Page 11 of 74

DATA SCHEMA AND DATA TYPES

Isn't MongoDB schema-less?

• MongoDB enforces no schema; documents don't have to use the same schema inside one collection.

• But that doesn't mean you can't use some kind of schema.

• Ultimately, we use both SQL & MongoDB world.

---

Data Types

• A document max size is 16MB

graph TD
    Text("Text - 5kB")
    Boolean("Boolean - True")
    Null("Null")
    Integer("Integer (32/64)")
    NumberLong("NumberLong (64)")
    NumberDecimal("Number Decimal (High Precision)")
    ObjectId("ObjectID - Object('')")
    ISODate("ISODate")
    Timestamp("Timestamp (1142)")
    Array("Array")
    Embedded("Embedded Document")

    Text --> Boolean
    Boolean --> Null
    Null --> Integer
    Integer --> NumberLong
    NumberLong --> NumberDecimal
    NumberDecimal --> ObjectId
    ObjectId --> ISODate
    ISODate --> Timestamp
    Timestamp --> Embedded
    Embedded --> Array

• Max level = 100

---

Page 12 of 74

Relations - Options

• Using Embedded documents (may have duplication)
• Using References (split data)

---

(a) One to One

• Person → Car
• If we use references, it may be fast with small data but not optimal with large data.
• So, use embedded documents.
• If an application isn't interested in whole document but only parts, we will use references.

---

(b) One to Many

• Thread → Q1, Q2
• Embedded is normal
• References to not hit 16MB limit

---

(c) Many to Many

• CA ↔ P1
• CB ↔ P2
• CC ↔ P3
• Embedded = lots of data duplication
• References = maybe a good idea

---

Page 13 of 74

Schema Validations

flowchart TD
    Document --> Collection --> ValidationSchema
    ValidationSchema -->|Accepted| Collection
    ValidationSchema -->|Rejected| Collection

---

---

Summary

I. In which format will you fetch data?
II. How often will you fetch & change your data?
III. How much data will you save (how big it is)?
IV. How is your data related?
V. Will duplicates hurt you (many updates)?
VI. Will you hit any storage limits?

---

Page 14 of 74

Explore Shell

Start MongoDB Services

(a)

sudo service mongod start

• Uses /var/lib/mongodb folder for db
• Uses /var/log/mongodb/mongod.log for logs

(b)

sudo mongod --dbpath <path> --logpath <log file path>

• Starts mongod server in specified folder

(c)

sudo mongod --config <path-file-config>

• Some settings to path & start it

---

Deeper Dive Into Inserts

(i) insertOne [more descriptive & returns id]
(ii) insertMany
(iii) insert [confusing]

• Bulk write in insertMany
• If it fails, it continues until failed
• Ordered Insert

---

Page 15 of 74

Ordered Inserts

• Every element you insert is processed standalone but if one fails it cancels the operation but it doesn't rollback.

• To change this default behavior, add this option:

  db.<collection>.insertMany([{}, {}, {}], { ordered: false })

---

Write Concern

• It describes the level of acknowledgment from MongoDB for write operations to an instance or multiple nodes.

  writeConcern: {
      w: <value>,
      j: <boolean>,
      wtimeout: <number>
  }

  • w: # of nodes
  • j: has server stored to disk
  • wtimeout: timeout error after specified time

• w: 0 → acknowledged = false (doesn't guarantee if write operation is done or not)

---

Page 16 of 74

Atomicity

• On a document level, a guarantee is that if it is done or not.

flowchart TD
    operation -->|Failure| rolled_back["Rolled back (i.e., Nothing is saved)"]
    operation -->|Success| saved["Saved as a whole"]

---

MongoImport

mongoimport -d <db-name> <json file path>
# <collection-name> --jsonArray --drop
# drop before inserting

---

Deeper Dive Into Reads

Access Current DB

db.myCollection.find({ eq: 3 })

• Access this collection

Range Filters

{ eq: { $gt: 30 } }

• Field, Operator, Value

---

Page 17 of 74

Operators

• Read

  • Query & Projection

• Update

  • Fields
  • Array

• Query Modifiers

  • Query Query Behaviour

• Aggregation

  • Pipeline Stage
  • Pipeline Operators

---

For READ

---

a) Query Selectors

1) Comparison

• $eq, $lt, $lte, $in (SQL: in)
• $ne, $gt, $gte, $nin (not in)

Example:

{ surname: { $in: [ "x", "y", "z" ] } }

---

Page 18 of 74

For embedded documents

{ 
  settings: {
    average: 6
  }
}

{ "rating.average": { $gt: 6 } }

2) Array

{ genre: [ "", "", "", "" ] }
{ genre: { $eq: "Drama" } }

• { genre: "Drama" }
• For arrays, it needs to find only one matching element.
• { genre: [ "Drama" ] }
• It gives exact equality.

3) Logical

{ $or: [ { "rating.average": { $lt: 4 } }, { "name": "Drama" } ] }

• $not
• $and
• $nor

---

Page 19 of 74

4) Element

• $exists (just check for field, fails for null values)

{ age: { $exists: true } }

• $type

{ phone: { $type: "number" } }

• "double"
• "string"

5) Evaluation

• $regex (regular expression)
  • Selects docs where values match a specified regular exp.

{ summary: { $regex: /musical/i } }

• Not quite efficient.

• $expr (later in aggregation module)

{ $expr: { $gt: [ "$return", "$target" ] } }

---

Page 20 of 74

6) Array

hobbies: [
  { frequency: 2, title: "Sports" },
  { frequency: 3, title: "Cooking" }
]

{ "hobbies.title": "Sports" }

• $size

{ "hobbies": { $size: 3 } }

• $all

{ "genre": { $all: ["action", "nullum"] } }

• $elemMatch

{ "hobbies": { $elemMatch: { title: "Sports", frequency: { $gt: 3 } } } }

---

Page 21 of 74

Cursors

• find()
• Millions/thousands of Docs

sequenceDiagram
    participant Client as Client (Cursor)
    participant DB as MongoDB Server/Database
    Client->>DB: Request Batch 1
    DB-->>Client: Data Batch 1
    Client->>DB: Request Batch 2
    DB-->>Client: Data Batch 2
    Client->>DB: Request Batch 3
    DB-->>Client: Data Batch 3

const dbCursor = db.<collection>.find()
dbCursor.count()
dbCursor.hasNext()
dbCursor.next() // Shows next doc in cursor
dbCursor // Shows 20 a/c to shell

dbCursor.forEach((doc) => printjson(doc));
print(tojson(doc));

• Sorting:

dbCursor.sort({ "rating.average": 1, run_time: -1 }) // Ascending, Descending
dbCursor.skip(10)
dbCursor.limit(5)

---

Page 22 of 74

Projection

By default, also including _id.

db.cname.find({ $y, $id: 0, names: 1 })

• $
• $elemMatch
  Projects the first element in an array that matches the specified $elemMatch condition.
• $slice
  Limits the number of elements projected from array.

---

Diving Deeper into Updates

db.<c>.updateOne({ $y }, { $set: { $z } })

• It can add if not present.

• It can overwrite only on array.

• $inc
  Add/subtract

{ $inc: { age: 2 }, $set: { age: 4 } }

• Increases if present, appends if absent.
• Error two operators on same field. → Error: two operators on the same field.

---

Page 23 of 74

$min

{ $min: { age: 5 } }

If age is more than 5, it will be updated.

$max

{ $max: { age: 3 } }

If age is less than 3, it will be updated.

$mul

{ $mul: { age: 5 } }

Multiplied by 5.

$unset

Removes the attributes.

{ $unset: { age: "" } }

$rename

Renames the attributes.

{ $rename: { age: "TotalAge" } }

Option: $upsert

If set to true, creates new document when no document matches the query criteria.
Default: false

---

Page 24 of 74

{ $z, $z, $object: true }

Array Operators

{ $z, $set: { "hobbies.$": { $z } } }

• Changes the entire document with $; only that document is updated.

{ "hobbies.$.newAttribute": true }
"hobbies.$[]"

• Changes all documents in the array.

{ $set: { "hobbies.$[el].goodFrequency": true } }

• Identifier:

arrayFilters: [{ "el.frequency": { $gt: 2 } }]

• Updates all array documents with match filters.

• $push, $each, $sort

{ $push: { hobbies: { ... } } }

---

Page 25 of 74

• $pull
  Removes all array elements that match a specified query.

{ $pull: { hobbies: ? } }

• $pop
  Removes first or last element in array.

{ $pop: { hobbies: 1 } } // last
{ $pop: { hobbies: -1 } } // first

• $addToSet
  Adds elements in array that are unique.

---

Diving Deeper into Delete Operations

db.<col-name>.deleteOne({ $y })
db.<col-name>.deleteMany({ $z }) // delete all entries in collection
db.<col-name>.drop()
db.dropDatabase()

---

Page 26 of 74

Retrieving Data Efficiently

INDEXES

db.<col-name>.find({ name: "max" })

• No Index → Collection Scan (COLLSCAN)
• With Index → Index Scan (IXSCAN)

flowchart LR
    COLLSCAN["Collection Scan (COLLSCAN)"]
    IXSCAN["Index Scan (IXSCAN)"]
    COLLSCAN -->|No index| Find
    IXSCAN -->|With index| Find
    IXSCAN -->|Ordered| Read Scalar Index
    subgraph Scalar Index
        direction TB
        A["Anna"]
        B["Max"]
        C["Munnu"]
        D["Max"]
    end

db.<col-name>.explain(), find(), update(), delete()
db.<col-name>.explain("executionStats").find()
db.<col-name>.createIndex({ <field-name>: 1 })
db.<col-name>.dropIndex({ <field-name>: 1 })

• Compound Index

createIndex({ <field>: 1, <field2>: 1 })

• Default Index

db.<col-name>.getIndexes()

There is one by default for "_id".

---

Page 27 of 74

Engineering Indexes

• Add unique

  { <field>: 1 }, { unique: true }

• Partial Index

  createIndex({ <field>: 1 })

  • Smaller index, used when only a set of data is used.
  • Partial filter expression, if granted 'match'.

Time to Live Index

createIndex({ <field>: 1 }, { expireAfterSeconds: 10 })

• Deletes document whenever it gets triggered.
• Works on only Date object.
• Only works on primary index.

---

Query Diagnosis & Query Planning

• explain()

---

Page 28 of 74

Efficient Queries

• Milliseconds processing time

Should be as close as possible:

$$\text{Ratio} = \frac{\# \text{ of documents examined}}{\# \text{ of documents returned}}$$

---

Covered Query

If an index has the total properties about the document, then there is no need to reach out to the collection and fetch one.

Example:

db.find({ name: "something" }, { _id: 0, name: 1 })

• Here name is in the index.
• So # of docs examined = 0
• Hence, this is called a covered query.

---

REJECTING plans in MongoDB

I. Looks at indexes which can help.
II. Use all available approaches to search for some threshold documents.
III. Whichever wins is the winning plan.

---

Page 29 of 74

IV. Now, MongoDB caches the winning plan to make future queries faster.

• Only for some future queries.

flowchart TD
    Threshold(Write threshold (1000))
    Reset(Index is reset)
    AddRemove(Other indexes are added/removed)
    Restart(Server restarts)
    Threshold --> Reset
    Threshold --> AddRemove
    Threshold --> Restart

Multi-key Index

• On an array of collection documents

I. createIndex({ hobbies: 1 })

• Array of "sports", "cook"
• find({ "hobbies": "sports" })
• Index scan is used.

II. createIndex({ addresses: 1 })

• Array of documents
• find({ "addresses.street": "main" })
• Coll scan is used.
• find({ "address": { streets: "Main" } })
• Index scan is used.

---

Page 30 of 74

• For a multi-key index, MongoDB creates multiple indexes.
  • E.g., array has 4 docs, then we have 4 indexes.

Text Index

createIndex({ description: "text" })

• Removes all the stop words and adds all the keywords.

find({ $text: { $search: "awesome" } })

• These are sparse indexes.

Sorting in Text Index

find({ $text: { $search: "awesome t-shirt" } }, { score: { $meta: "textScore" } })
sort({ score: { $meta: "textScore" } })

Configuration

createIndex({ name: "text" }, { default_language: "english", weights: { name: 1, desc: 10 } })

---

Page 31 of 74

Note:
Indexes lock the DB while getting created.
So, for production environment:

set background: true

---

Page 32 of 74

Working with GeoSpatial Data

GeoJSON object

{ name: "?", location: { type: "Point", coordinates: [longitude, latitude] } }

GeoJSON Query

find({ location: { $near: { $geometry: { type: "Point", coordinates: [longitude, latitude] }, $maxDistance: ?, $minDistance: ? } } })

Note: Requires geospatial index.

createIndex({ location: "2dsphere" })

Find locations inside a given polygon

find({ location: { $geoWithin: { $geometry: { type: "Polygon", coordinates: [[p1, p2, p3, p4, p1]] } } } })

---

Page 33 of 74

(iii) Find places within a certain radius

• Find location:

  db.geoWithin.$centerSphere = [ [ <long, lat> ],
    1 / 6378
    (km)
  ]

  • Where 6378 is the Earth's radius in kilometers.
• $near sort results also.

---

Page 34 of 74

MongoDB Security

• Authentication & Authorization
• Transport encryption → Transport Encryption
• Encryption at Rest
• Auditing
• Server & Network config → Server & Network Configuration
• Backup & Updates

Developer Concern vs Server Admin Concern

---

Authentication & Authorization

• Example:
  • You are employed, therefore may enter office.
  • You are employed as an accountant and can do different actions, not software dev.

---

Role Based Access Control

flowchart LR
    A[Not user of Application] --> B[MongoDB User<br/>Data Analyst, etc.]
    B -->|Login with Username & Password| C[Logged in, but no rights to do MongoDB Server]
    C --> D[Shop DB]
    C --> E[Blog DB]
    C --> F[Admin DB]
    G[Roles/Privileges] --> H[Resources/Action]

---

Page 35 of 74

Why roles?

• Different types of DB users:

flowchart TD
    updateUser --> Roles
    createUser --> Roles
    Roles --> Database

How?

1. $ sudo mongod --auth

   To start mongodb server in authentication mode.

2. $ mongo -u <username> -p <password> --authenticationDatabase admin

   Before that, we need to have one user.

---

Page 36 of 74

Well, MongoDB has a local host connection which allows to add one user, and this user is then allowed to create more users.

For this:

$ use admin
$ db.createUser({ user: "max", pwd: "sk", roles: [ "userAdminAnyDatabase" ] })
$ db.auth("max","sk")
$ show dbs // now possible

Built-in Roles

---

Page 37 of 74

Example

For shop database:

$ use shop
$ db.createUser({ user: "bud", pwd: "...", roles: ["readWrite"] })

$ use admin
$ db.logout()
$ use shop
$ db.auth("...", "...")
$ db.products.find().pretty()

Example

$ mongo -u --- -p --- --authenticationDatabase shop
$ use shop
$ db.products.find()

Update User

• Login as admin on admin DB

$ use shop // because user is in shop
$ db.updateUser("nash", { roles: ["readWrite"] }) // for current DB
$ db.updateUser("nash", { roles: [{ role: "readWrite", db: "test" }] })

---

Page 38 of 74

II. Transport Encryption

flowchart LR
    Client -->|Encrypted| MongoDBServer
    subgraph Client
        App
        MongoDB
    end

---

Page 39 of 74

Performance, Tolerance & Deployment

What influences performance?

• Efficient Queries
• Indexes
• Fitting Data Schema
• Hardware & Network
• Sharding
• Replica Sets

Developer/Admin Concern vs System Admin Concern

---

Capped Collections

db.createCollection("collection_name", {
  capped: true, size: 10000, max: 3
})

• Always sorted in these collections by id.
• Remove first collection on size limit hit.

---

Replica Sets

flowchart TD
    Client --write--> MongoDBServer
    MongoDBServer --write--> PrimaryNode
    PrimaryNode --Async Replication--> SecondaryNode
    PrimaryNode --Async Replication--> SecondaryNode

---

Page 40 of 74

Why Replica Set Read?

flowchart TD
    MongoDBServer -->|Read + Write| PrimaryNode
    MongoDBServer -->|Read from here| SecondaryNode

• Backup/fault tolerance
• Improve Read Performance
  • Read requests can go to all secondary nodes

---

Sharding (Horizontal Scaling)

flowchart LR
    MongoDBServer
    subgraph Shards
        direction LR
        A[Shard] B[Shard] C[Shard] D[Shard] E[Shard]
    end

• Splits data (chunks of data)
• Data is distributed (not replicated) across shards.
• Queries can run across all shards.

---

Page 41 of 74

flowchart TD
    Client --> Mongos[Router]
    Mongos --> Shard1
    Mongos --> Shard2
    Mongos --> Shard3
    Shard1 -->|Shard key| Documents
    Shard2 -->|Shard key| Documents
    Shard3 -->|Shard key| Documents

Queries & Sharding

find()

• Option 1: Operation does not contain shard key
  • Broadcast to all shards
• Option 2: Operation does contain shard key
  • Directly send to right shard

---

What goes to deploy MongoDB servers?

• Manage Shards
• Secure User/Auth setup
• Protect Web Server/Network
• Manage Replica Sets
• Encryption (at transport/at rest)
• Regular Backups
• Update Software

---

Page 42 of 74

Transactions (Uses session)

const session = db.getMongo().startSession();
session.startTransaction();

const v1 = session.getDatabase("<db_name>").<col_name>
const v2 = session.getDatabase("<db_name>").<col_name>

// After this
v1.deleteOne()
v2.deleteMany()

session.commitTransaction() // changes are done (either succeed or rollback)
session.abortTransaction() // cancels it!

• Added as a todo from MongoDB.

---

Page 43 of 74

Aggregation Framework: (Alternative to find)

• Retrieving data efficiently & in a structured way.

flowchart TD
    Collection --> match
    match --> sort
    sort --> group
    group --> project
    project --> Output

• Every stage receives the output of previous stage.

db.<colname>.aggregate([])

• Takes an array of operations.

.aggregate([
    { $match: { gender: "female" } },
    { $group: { _id: { state: "$location.state" }, totalPersons: { $sum: 1 } } }
])

---

Page 44 of 74

Notes:

• $field → refers to value
• field → refers to field

{ $sort: { totalPersons: -1 } },
{ $project: { _id: 0, gender: 1, fullname: { $concat: [ "$name.first", " ", "$name.last" ] } } }

• $concat can receive more complex statements.

Example:

{ $toUpper: "$name.first" }
{ $toUpper: { $substrCP: [ "$name.first", 0, 1 ] } }
{ $substrCP: [ "$name.first", 1, 2 ] }
{ $subtract: [ { $strLenCP: "$name.first" }, 1 ] }

---

Page 45 of 74

Example: Project for a GeoJSON object

{ $project: {
    _id: 0,
    name: 1, // for next project
    email: 1,
    location: {
        type: "Point",
        coordinates: [
            "$location.coordinates.longitude",
            "$location.coordinates.latitude"
        ]
    },
    $convert: {
        input: "$location.coordinates.longitude",
        to: "double",
            }
}}

---

Page 46 of 74

Example: Project for a DOB

{ $project: {
    _id: 0,
    birthdate: {
        $convert: {
            input: "$dob.date",
            to: "date",
            onError: null
        }
    }
}}

---

$group vs$project

---

Array operations in Aggregation module

Push every hobbies into all hobbies grouped by age:

db.<colname>.aggregate([
    { $group: { _id: { age: "$age" }, allHobbies: { $push: "$hobbies" } } }
])

---

Page 47 of 74

II. $unwind

• Runs an array to $unwind and flattens the array into multiple docs.

• Group hobbies by age is like:

db.<colname>.aggregate([
    { $unwind: "$hobbies" },
    { $group: { _id: { age: "$age" }, allHobbies: { $push: "$hobbies" } } }
])

• We can use $addToSet to avoid duplicate values.

---

III. $slice

• Returns slice of an array.

Example:

{ $project: { _id: 0, examScore: { $slice: ["$examScores", 2] } } }

• Any array, length to get. If it is -1, it takes from last.

[ array, start, length ]

---

Page 48 of 74

IV. $size

• Returns size of an array.

V. $filter

• Filter out options from an array.

Example:

{ $project: { examScores: { $filter: { input: "$examScores", cond: { $gt: [ "$$this.score", 60 ] } } } } }

---

Find highest exam score of each person in array

Array looks like this:

[
    { "_id": ..., "examScores": [ { difficulty: 5, score: 75 }, { difficulty: 3, score: 40 } ] },
    { ... another ... }
]

After processing:

[
    { "_id": ..., "examScores": 75 },
    ...
]

---

Page 49 of 74

MongoDB Advanced Aggregation Stages

Example Aggregation Pipeline

db.<coll>.aggregate([
  { $unwind: "$examScores" },
  { $project: { _id: 1, score: "$examScores.score", name: 1 } },
  { $sort: { score: -1 } },
  { $group: { _id: { _id: "$_id", name: "$name" }, max_score: { $max: "$score" } } }
])

Some Additional Stages

1. Bucket
2. Bucket Auto

Example: $bucketAuto

db.<coll>.aggregate([
  {
    $bucketAuto: {
      groupBy: "$dob.age",
      buckets: 5,
      output: {
        numberOfDocs: { $sum: 1 },
        averageAge: { $avg: "$dob.age" }
      }
    }
  }
])

• Helps in visualization of data.

---

Page 50 of 74

More Aggregation Stages

3. Limit Stage

   { $limit: 10 }

   Get only first 10.

4. Skip Stage

   { $skip: 10 }

   Skip first 10. (Depends on order)

5. Out Stage

   • Takes out result and writes into the new collection (already existing/new).

   { $out: "transformedData" }

---

Page 51 of 74

Working with Numeric Data

• Mongo shell is written in JavaScript
  So, Int32 is stored as Float64.
  • If working from NodeJS driver, same thing happens.
• But, from Python Driver, it differentiates Int32 and Float64 because Python differentiates them.
• So, value written in MongoDB depends on client you are using.

Number Types in MongoDB

Int32

db.person.insertOne({ age: NumberInt(219,45) })

or

db.person.insertOne({ age: "219,458" }) // Pass as string

• Stored as 219
• Greater (2^{32}-1) → overflows

Int64

{ age: NumberLong("17") }

• Pass as string.

---

Page 52 of 74

• During maths, be sure of typecasting.
• All numbers are Float64 by default.
• High precision Doubles:

{ a: NumberDecimal("0.345") }

---

Page 53 of 74

From Shell to Driver

Shell vs Driver

Connecting Driver of NodeJS

const MongoClient = require('mongodb').MongoClient;

MongoClient.connect(URL, (err, client) => {
  if (err) {
    return console.log(err);
  }
  console.log('Connected to Server');
  const db = client.db('TodoAPP');

  client.close();
});

ObjectId

• Time Stamp + MachineId + Random Value = _id
• So, _id.getTimestamp() returns time.

---

Page 54 of 74

Collection Methods

db.collection('<coll-name>').findOne()
db.collection('<coll-name>').find({}) 
db.collection('<coll-name>').insertOne()
db.collection('<coll-name>').insertMany()
db.collection('<coll-name>').updateOne()
db.collection('<coll-name>').updateMany()
db.collection('<coll-name>').deleteOne()
db.collection('<coll-name>').deleteMany()

• This gives a cursor.
  • .toArray()
  • .forEach()

findOneAndUpdate({ ... }, { $set: { ... } }, options, (err, doc) => { ... })
findOneAndDelete({ ... }, (err, doc) => { ... })

• Returns doc as well.

---

Driver to ORM/ODM

Object-Relational Mapper → For SQL/relational DB

Object-Document Mapper → For NoSQL DB

• Mongoose is an ODM that provides a straightforward & schema-based solution to model your application data on top of MongoDB's native driver.
• Built-in type casting, validation, query building, hooks, etc.

var mongoose = require('mongoose');
var Schema = mongoose.Schema;

var blogSchema = new Schema({ ... });

---

Page 55 of 74

Defining Mongoose Schema

{
  title: { type: String, required: true },
  age: Number,
  dob: { type: Date, default: Date.now }
}

module.exports = mongoose.model('Product', productSchema);

• Gets converted to lower case & added plural form.

---

Schema Types

Note:
Mongoose kind of "type" casting, it may be a trouble.

---

Instance Methods

blogSchema.methods.<new-method-name> = function(cb) { ... }

or

blogSchema.method('new-method-name', function() { ... })

blogSchema. → blogSchema.method

---

Page 56 of 74

Static Methods

• Similar way declaration.

CRUD

I. Create

var Tank = mongoose.model('Tank', tankSchema);
var document = new Tank({ ... });

document.save((err) => {
  ...
});

or

Tank.create(doc, (err, doc) => { ... });

or

Tank.insertMany([ ... ], (err, docs) => { ... });

II. Read / Find

Tank.find({ ... }).where({ ... }).gt(5).exec((err, docs) => { ... });
Tank.findById(id);
Tank.findOne({ ... });
Tank.find({ fields: 'name age' }, (err, docs) => { ... });

---

Page 57 of 74

III. Update

Tank.updateOne({ ... }, { ... }, { runValidators: true }, (err, doc) => { ... });

IV. Delete

Tank.deleteOne({ ... }, (err) => { ... });

---

Page 58 of 74

FRONTEND STORAGES

I. Cookie

• It is a small piece of data that a server sends to user's web browser.
• The browser sends it back with next request to same server.
• So, we can say it remembers stateful information for the stateless HTTP protocol.
• They are used for:
  • Session Management
    • (See NodeJS session cookies)
  • Personalization
  • Tracking

General Syntax

Set-Cookie: <cookie-name>=<cookie-value>

---

II. IndexedDB

• It is a low-level API for client-side storage of significant amounts of structured data, including file/blobs.
• It uses indexes to enable high-performance searches of data.

Features:

• It is a transactional database system.
• It is for JavaScript-based object-oriented databases.
• Stores JS objects, files, blobs, etc.
• Operations performed using IndexedDB are done async so as not to block apps.
• Some storage limits depend on browser.

---

Page 59 of 74

III. Web Storage API

• It provides mechanisms by which browsers can store key/value pairs in a much more intuitive way.
• There are two mechanisms within web storage:
  • a) Session Storage:
    Maintains a separate storage area for each given origin that's available for the page open (reload, restore).
  • b) Local Storage:
    Same thing, but persists even when browser is closed and reopened.

Example

localStorage.getItem('key')
localStorage.setItem('key', 'value')
localStorage.removeItem('key')
localStorage.key(n)
localStorage.clear()

Note:
window.x = x

---

IV. Cache API

• It provides a storage mechanism for request/response object pairs that are cached.
• They are part of ServiceWorker life cycle, but it is exposed to windowed scopes as well.

---

Page 60 of 74

Service Worker (Optional)

• A JS script that gets registered with the browser, stays registered even when offline.

Use Cases

• Caching assets & API calls
• Push Notifications
• Background data sync (x)
• Used in progressive web apps

---

Security

• The act/practice of protecting websites from unauthorized access, use, modification, destruction, or disruption (e.g. hackers).

XSS: (Cross Site Scripting)

• Code injection attack that allows an attacker to execute malicious JS in another user's browser.
• It can't be done directly. Instead, attacker exploits a vulnerability in website that the victim visits, which in turn delivers malicious JS to victim.

How malicious JS is injected?

• Website with comments

<script>getCookie()</script> <!-- sent to their server -->

• User: Hey
• Victim: I am innocent.

Result:
Cookies, keylogging, phishing, form innocent.

---

Page 61 of 74

Types of XSS

1. Persistent
2. Reflected
3. DOM-based (invisible to server)

How to prevent it?

• Encoding – user input to data only, not as code.
• Validation – filter user input.

---

State of Security

Injections

• Code inside other code.

Example: SQL Injection

OR 1=1 --  (comment)

→ Read all data

; DROP TABLE users; -- drop table

Example in form fields

<img src="/" onerror="alert('boom')" />

• If done via p.innerHTML

Better:

var textNode = document.createTextNode(input);
h.appendChild(textNode);

---

Page 62 of 74

How to fix them?

1. Sanitize input
2. Parameterize queries
3. knex.js or other ORMs
4. check express-validator

---

3rd Party Libraries

• Use:
  • npm audit (npm package) # audit packages
  • snyk test # audit node_modules directory

---

Logging

• winston or morgan

---

XSS & CSRF

CSRF: Cross-site request forgery

Attack that forces an end user to execute unwanted action on a web application in which they're currently authenticated.

Example

<img src="https://www.example.com/index.php?action=delete&id=123" />

---

CORS

• It is a mechanism which aims to allow requests made on behalf of you & at the same time, block some requests made by rogue JS & is triggered whenever a HTTP request is made to:
  • a different domain
  • a different sub-domain
  • a different port
  • a different protocol (HTTP, HTTPS)

Example

flowchart LR
  HTML[HTML Template]
  POST[POST request to mybank.com]
  HTML --> POST

Headers

• Access-Control-Allow-Origin

• Credentials

• Headers

• Methods

• Some headers to do all this BS

---

Page 63 of 74

Storing Information in Browser

1. Cookies
2. Session

Cookies in NodeJS (stored on client side)

sequenceDiagram
  participant User
  participant Frontend
  participant Cookie
  participant Server

  User->>Frontend: Interact (view)
  Frontend->>Cookie: Set
  Frontend->>Server: Request (include cookies)
  Server->>Frontend: Set via Response Header

Why Cookies?

• If you used some global variable to store some info, it will be shared across every user for every request, so we use cookies.

res.setHeader('Set-Cookie', 'loggedIn=true')

• Here, a cookie can be manipulated by client by setting to true/false.

Some options to cookies

• Secure
• HTTP Only
• Max Age
• Expires In
• [See documentation for more]

---

Page 65 of 74

Sessions in Node.js (stored on server side)

flowchart TD
  User -- request --> Frontend[Frontend Views]
  Frontend -- Cookie --> Cookie
  Cookie -- Associated with user/client via cookie --> Server[Server (Node App)]
  Server -- session --> Session
  Session -- session storage --> Database

• Session: Mechanism for managing user data on the server side.
• Session Storage: Typically in a database.

Using express-session package for managing sessions

• Options:
  1. cookie
     • Only session ID is stored in cookie; session data is stored on server side.
     • Options:
       • domain
       • expires
       • maxAge
       • httpOnly
       • sameSite
       • secure
       • path
  2. resave = false
     • force the session to be saved back to session store even if session was never modified during the request → Forces session to be saved back even if unmodified.
  3. saveUninitialized = false
  4. secret
     • Used for signing the session ID cookie.

---

Page 66 of 74

Session Store Example

app.use(
  session({
    secret: "",
    resave: false,
    saveUninitialized: false,
    cookie: {}
  })
);

Note: Session data for every user is stored in memory, which will be trouble → troublesome for millions of requests.

Use session stores, e.g.

• connect-mongodb-session
• connect-mongo
• connect-redis

var MongoDBStore = require('connect-mongodb-session')(session);

var store = new MongoDBStore({
  uri: DB_URL,
  collection: '<col-name>'
});

• Add store option to abstract session.

---

Page 67 of 74

User Authentication

flowchart TD
  User -- View/Add/Delete/Logout in UI --> Server
  Server --> Database

  User -- login request --> Server
  Server -- session info --> Session
  Session -- Stores info that user is authenticated --> Database

  User -- Cookie --> Server
  Server -- Request restricted resource --> Session

Storing password in the database

bcrypt.genSalt(12, (err, salt) => {
  bcrypt.hash(password, salt, (err, hash) => {
    console.log(hash);
  });
});

Comparing Passwords

bcrypt.compare(~~body.password~~ → inputPassword, actualPassword, (err, result) => {
  if (err) return;
  if (result === true) return true;
});

---

Page 68 of 74

Sending Mails

sequenceDiagram
  participant NodeServer
  participant MailServer
  participant User
  NodeServer->>MailServer: Send mail
  MailServer->>User: Send mail

• Mail server isn't an easy task.
• nodemailer package handles mail services (needs 3rd party service also).

Resetting User Password

• Generate random token → send to email when user requests.
  • Save it in user (field: email).
  • Add expiry.
  • Check for valid token.
  • Reset authenticated password if valid.

Validating User Input

• May use mongoose, but server-side validation is fast.
• Check for express-validator.

Error Handling

• Use try-catch, Sync/Async.

---

Page 69 of 74

REST API

Representational State Transfer

(Transfer Data Instead of User Interface)

The different data formats are:

REST Principles

• Uniform Interface: Closely defined API endpoints with clear req & res data structure.
• Stateless Interactions: Every req is handled separately.
• Cacheable: Servers may set caching headers to allow the client to cache responses.
• Client-Server: Server & client are separated; client not concerned with persistent data storage.
• Layered System: Server may forward requests to other APIs.
• Code on Demand: Executable code may be transferred from server to client.

---

Page 70 of 74

Fixing CORS Concerns

app.use((req, res, next) => {
  res.setHeader("Access-Control-Allow-Origin", "*");
  res.setHeader("Access-Control-Allow-Methods", "GET, POST");
  res.setHeader("Access-Control-Allow-Headers", "Content-Type, Authorization");
  next();
});
app.use(cors());

• Also, use the cors package.

Authentication in REST APIs

Rule 1: Store passwords in hashed forms.

• Use bcryptjs (Pure JS implementation of bcrypt C++ library).

For sync (not recommended):

var bcrypt = require("bcryptjs");
var salt = bcrypt.genSaltSync(10); // or use a number
var hash = bcrypt.hashSync("<pass>", salt);
// check password
bcrypt.compareSync("<pass>", hash); // Returns true or false

---

Page 71 of 74

For Async

bcrypt.genSalt(10, (err, salt) => {
  bcrypt.hash(password, salt, (err, hash) => {
    // ...
  });
});

Auto-gen a Salt & Hash

bcrypt.hash(password, salt-number, (err, hash) => {
  // ...
});

Rule 2: Generate JWT tokens.

var data = { /* ... */ };
var token = {
  data,
  hash: SHA256(JSON.stringify(data) + "somesecret").toString()
};
var resultHash = SHA256(JSON.stringify(token.data) + "somesecret").toString();

if (resultHash === token.hash) {
  // Valid
}

---

Page 72 of 74

JWT

• jsonwebtoken is the npm package.

To sign:

jwt.sign(object-data, secret); // returns a token

• This token is a combination of:

To verify:

jwt.verify(token, secret);
// returns data if succeeded, else returns error

---

Page 73 of 74

Network Performance

Images

• PNG/JPEG: tinypng.com
• JPEG: jpeg-optimizer.com
• Remove meta data

HTML/CSS/JS

• Minify them
• Make them one

Critical Render Path

flowchart LR
  DOM --HTML--> CSSOM --(DOM content loaded)--> RenderTree --> Layout --> Paint
  DOM --CSS--> CSSOM
  DOM --JS--> JS
  JS -.-> RenderTree

• <script async>: Runs with another thread

• <script defer>: Runs after script downloaded in background

• Find page speed at:

  • "page speed insights"
  • "webpage test test"

---

Page 74 of 74

Backend Performance

CDN: (Content Delivery Networks)

graph TD
  OriginServer --Content--> CDN1
  OriginServer --Content--> CDN2
  CDN1 --Content--> User
  CDN2 --Content--> User

• It does caching:
  • Improves page load
  • Higher traffic load handled
  • Blocks bots/spammers
  • Can prevent DDoS

GZIP (Google created Brotli)

• Brotli is 20% faster than gzip.
• In node.js, it is done by compression.

DB Scaling

1. Identify inefficient queries
2. Increase memory
3. Vertical scaling (Redis/Memcached)
4. Sharding
5. More databases (Horizontal scaling)
6. Database Type (more specific on application, e.g., MongoDB: key-value, SQL: relational)

Load Balancing

• (Load it on Nginx)

---

References & Related Topics

• Express.js Documentation
• MongoDB Manual
• Node.js Documentation
• Related: REST APIs, Mongoose, ORM/ODM, Database Indexing, Server-Side Rendering, Authentication.