Logbook

Things I've learned
along the way.

Technical deep-dives and practical notes from 15+ years working with databases, infrastructure, and software engineering.

110Entries
8Topics
15+Years

MongoDB

NoSQL database design, operations, replication, and sharding.

26 entries
  1. Introduction to MongoMongo database is an open source , high performing , schema - free, non relational and document oriented database which provides highest level of availability1 min read
  2. Starting Database in MongoTo start the mongo database server on the linux machine with specific parameters for the database to start with we can use a config file , generally named as1 min read
  3. JSON & BSONJavaScript Object Notation (JSON) is an open, human and machine-readable standard that facilitates data interchange, and along with XML is the main format for1 min read
  4. CRUD in MongoCRUD corresponds to Create,Read,Update and Delete.Like any other database technology, mongo also supports CRUD operations on it by the use of its various2 min read
  5. Mongo Basic QueriesMongo database supports a data driven style of deceleration in the form of JSON and BSON.To run the various queries in mongo , first start the mongo database2 min read
  6. Mongo OperatorsMongo database provides various operators to support the querying on the mongo shell. These are some of the operators used: Querying : 1. $gte,$gt,$lte,$lt4 min read
  7. Mongo IndexesIndex in a database is what bookmark is to a book, it helps the fast traversal and retrieval of documents through indexing of each document of a collection.3 min read
  8. Explain Plan in MongoExplain plan can be used on a collection to get an explainable object. db.test1.explain().find({name:"hello"}).sort({empid:1}) { "queryPlanner" : {3 min read
  9. Mongo Import & CursorsMongo database supports data to be imported to the database in the form of csv,tsv and json file formats.Below command can be used to import the file3 min read
  10. Administrative Commands in MongoThe following are some of the commands that can be used for database administration for mongo. 1. getLastError db.getLastError() E11000 duplicate key error5 min read
  11. Profiler in MongoProfiler can be used to setup logging on the mongo instance. Events captured by the profiler: CRUD Administrative Operations Configuration Operations2 min read
  12. Wired Tiger in MongoWired tiger is a pluggable storage engine with following features: Document level locking Compression Fulfills some pitfalls of MMAPv1 Performance gain We can2 min read
  13. MMAPv1Storage Engine determines data file format & format of indexes and are key to performance for the mongo database.It is the middletier between the database and1 min read
  14. GridFS in MongoGridFS in mongo can be used to store and retrieve huge files of type image,videos,audios etc in the mongo database in the form of collection by considering the2 min read
  15. Mongo ReplicationReplication in any database is critical for setting up a secondary instance for the database instance.In mongo replication serves the purpose of : HA - High2 min read
  16. Mongo Replication SetupFor a statement based replication oplog file is used to apply all the transaction on the primary node to the secondary node.The default value for oplog3 min read
  17. Arbiters,Oplog,Write Concern in Mongo ReplicationRead Preference a.k.a “slaveOk”. In shell - rs.slaveOk() In driver – read preference Secondary can have potentially stale data due to lag in write from primary5 min read
  18. Sharding in MongoSharding is the method of data distribution in a mongo database environment where the data can be partitioned into different servers on the basis of "shard6 min read
  19. Sharding Setup in MongoTo setup mongo sharding environment we should be ideally having the below setups A Config Databases Replica Set of 3 or more members Multiple Shard Database5 min read
  20. JS Script for Shard Setup in MongoInstead of setting up each shard replica set and adding them to shard in the mongos shell we can use the below js script and load it to mongos to add mongo2 min read
  21. Security in MongoBelow table would give in various security features provided by Mongo:3 min read
  22. Mongo Installation on RHEL VMTo install mongo on a red hat VM , download the rpm files from mongo website according to the RHEL version you are using and the mongo version you want:1 min read
  23. Disaster Snapshot Recovery in MongoSay for in a database maintenance activity we drop a collection by mistake, we can restore to the last state just before dropping the database using oplog file1 min read
  24. Ops Manager in MongoOps Manager is a monitoring and administering web based tool provided by Mongo which would provide a single click provisioning for scaling,upgrades and1 min read
  25. Unix InstallationIn unix system installation can be done in more than one ways.For installing mongo in unix/linux systems follow the below steps using apt-get command use the1 min read
  26. Windows InstallationFor installing Mongo DB in windows server follow the below steps : 1. download the latest version of mongo from the mongo db website1 min read

MySQL

MySQL architecture, configuration, replication, and performance tuning.

22 entries
  1. Introduction to MySQLMySQL is an open-source relational database management system that works on many platforms. It provides multi-user access to support many storage engines and1 min read
  2. Basic Prompt CommandsTo login to a particular database use the below command: mysql -u root -p <dbname To connect to mysql server installed on a remote machine mysql -u root -p1 min read
  3. Basic Queries in MySQLAfter installing MySQL-Workbench , in the workbench or by opening mysql prompt in terminal use the below queries for different operations on the database 1. To1 min read
  4. Storage Engines in MySQLMySQL supports various storage engines to service the database server and perform the database activities. Each storage engine is unique and provides a2 min read
  5. Tablespace in MySQLIn MySQL table space can be created to store tables of the MySQL server. If we don't create a tablespace of our own the default tablespace for any new table1 min read
  6. Point in Time Recovery in MySQLPoint in Time recovery is the recovery mechanism where we can recover a lost transaction back to our MySQL server.The transaction here might be a database or a2 min read
  7. MySQL PerconaPercona is a third party tool used for taking physical backup of the entire MySQL Server to a backup path. Installation of Percona 1. Download the percona rpm1 min read
  8. Dump and RestoreThe logical dump of all database can be done using mysqldump utility as below: mysqldump -u root -p --all-databases alldatabases.sql The logical dump of a1 min read
  9. Galera ReplicationGalera replication is a third party open source replication utility for MySQL.Follow the below steps for setting up galera replication on all the three nodes1 min read
  10. Group Multi Master ReplicationGroup multi master replication is a master-master replication setup in MySQL. Follow the below steps to configure group multi replication: 1.Considering 33 min read
  11. NDB ClusterNDB Cluster or Next DB Cluster in MySQL is cluster management in MySQL where multiple data nodes with multiple servers can be configured for setting up2 min read
  12. GUI ClientsThere are very GUI clients are present in market to manage and administer the MySQL databases.These are DBA control consoles which provide easy access to1 min read
  13. Resetting root password for MySQLIn MySQL administration ,root access is critical to perform various administration tasks on the database. By any chance if the root password is lost or1 min read
  14. Starting Multiple Instances in MySQLWe can start multiple MySQL instance in the same server by running them in different ports. To configure multiple instances, 1. add the below entries in1 min read
  15. Command Line ClientsCommand Line tool benefits Quick Specific Low overhead Scriptable Schedulable Combinable General Syntax: command --opt1 --opt2=value[target] Names usually4 min read
  16. MySQL ArchitectureMySQL follows a client server architecture where the mysqld programs which are the core programs of the database system act as the server and various agent4 min read
  17. MySQL Configuration & SetupThis blog will cover the various configuration and setups that can be performed over a mysql server for its operations. Server command line options We can2 min read
  18. MySQL InstallationWhat is Installed bin directory ( and sbin in some ) = contains server prgms (mysqld and others) data directory( contains db directory ) innoDB tablespace file2 min read
  19. MySQL Resource UsageGlobal Memory Global buffers and caches are allocated on startup of mysqld and shared by all sessions.These can be general or storage engine specific.These are4 min read
  20. MySQL SecuritySecurity for a database could be at a physical level,network level,OS level,File System level or at User account level.The mysql.user table contains account7 min read
  21. MySQL ServerMySQL Server has different types of distribution starting from the Original MySQL from Oracle who are the current owners of MySQL.In 2010 Oracle had merged3 min read
  22. ReplicationFollow the below steps to configure MySQL Replication in the Linux servers : 1. For VM systems firewall has to be disabled to enable replication connections3 min read

Progress/OpenEdge

Progress OpenEdge DBA essentials — architecture, administration, and scripting.

40 entries
  1. Progress Openedge Database OverviewThis blog will walk you through the Progress OpenEdge Database basics and help you get an idea you need essentially to become a Progress Database5 min read
  2. OpenEdge Installation WindowsPrerequisites: DBA to make sure Java already installed and we have progress Setup downloaded in the server with serial number and control numbers, and2 min read
  3. OpenEdge Installation Unix/LinuxThese are the steps for installing progress openedge RDBMS in a nix OS: Installation Steps for Progress OpenEdge Product: Pre-requisite: DBA to make sure Java4 min read
  4. OpenEdge Database CreationSince we are done with installation of the Progress Openedge product , now we are free to play around with the features of Openedge and explore the opertaions2 min read
  5. Openedge Database ArchitectureNow since we are done with the history and installation of the Openedge Database. Let's move into the technical aspect of the OpenEdge Database,which i guess3 min read
  6. Openedge Storage AreasStorage areas are identified by their name. Each Progress database must contain three specific storage areas, and may contain several optional storage2 min read
  7. Database BlocksWe have now covered the Database architecture for an Openedge database and its various components such extents , storage areas etc.Now lets discuss the4 min read
  8. Database Structure Description File (.st )The most basic procedure to create a database is using the prostrct utility. To create a database using prostrct, we would require a structure description file2 min read
  9. OpenEdge Indexes and Index UtilitiesWhat is an index ? Database index works like a book index. To look up a topic, you scan the book index, locate the topic, and turn to the pages where the4 min read
  10. Openedge Dump and LoadThe Openendge RDBMS database, as we know ,would comprise mainly of tables and the data contained it. These table structures and the data containing in it could4 min read
  11. Openedge Database AuditingAuditing goal can be achieved through below steps: 1. Enabling Audit in a DB. 2. Implementing policies. 3. Generating reports. ENABLING AUDIT : 1. Create a4 min read
  12. User Management in Openedge DatabaseThe user management of a progress Openedge database refers to the user creation on the database with adequate privileges intended for the particular user.In5 min read
  13. Openedge Replication Monitoring StatusWhen "dsrutil dbname -C status" is used, the "Status" codes are returned When "dsrutil dbname -C status -detail " is used, the "Detail Status" codes are2 min read
  14. Steps for OpenEdge Replicationbelow steps for OS Copy Replication 1.Copy the source.repl.properties file from dlc/properties path and copy the the DB path ,giving it the db name eg:2 min read
  15. Openedge Replication Monitoring and commandsThe replication process is a real time syncing of the master and slave database. Any discrepancies or issues can lead to huge lag betweeen data updated in3 min read
  16. Openedge Appserver, Webspeeds, NameServer and Admin ServersAppserver Environment provides a foundation for a flexible and extensible application infrastructure. OpenEdge Application Server supports an open,4 min read
  17. Web Service Setup in ProgressFollowing are the steps to follow to setup web service in a progress environment: 1. First we would require a tomcat to deploy web service application, so copy1 min read
  18. OpenEdge Management(OEM) or Management Console/OpenEdge ExplorerIn windows version of progress openedge, the product package comes with an extra product which is not available in unix version of the openedge products -1 min read
  19. Setting up a Database to Monitor in OEMAfter doing initial configuration and setup , a OEM dashboard on login will look like as below showing the admin server and the various resources under the1 min read
  20. Progress 4GL Development BasicsABL (Advanced Business Language). Earlier known as Progress 4GL. Even if you are a progress database administrator, having knowledge of progress 4GL language1 min read
  21. Shell Script - Progress Disconnect UserScript : Disconnect user from database Script Name: disconnect.sh Script Function: Disconnect users from databases using "proshut" Required Files: 1.3 min read
  22. NameServer,AppServer,AdminServer,Webspeed Configuration and CommandsSince we are done with the theoritical part of the various entities , lets see how we use them in a real work scenario for a progress DB environment. Admin2 min read
  23. Openedge Database FilesOnce you create/start a database various db related files would be created.These files would serve for the operation and functioning of the databases. Below1 min read
  24. Openedge Database ReplicationOpenedge Replication is a feature/product from Progress that provides real-time synchronization between two or more databases. There is a replication server1 min read
  25. Openedge Database ReportsOpenEdge provides us a variety of utilities to pull out reports related to the database and which we can use to analyse and plan how the database7 min read
  26. OpenEdge Database Startup ParametersOnce you have created a database , the database would be available in a single user mode, i.e. only one user would be able to access the database at a time.9 min read
  27. OpenEdge Interview QuestionsProgress Interview Questions: 1. Which version of progress you have worked on ? diff and Features 2. How you are monitoring your database? 3. Have u installed2 min read
  28. Openedge Schema TablesWhat are schema tables?  Schema tables are metadata: i.e data about data  These are real tables, stored in the database.  It describe what is stored in the3 min read
  29. Operating System ResourcesProgress uses several operating system resources, such as shared memory, memory locks, and processes.You can plan Progress database operations more effectively2 min read
  30. Restricting blank user id login to progress databaseSecurity is a major aspect for any database,it is required to have a user/pass credential validation for every connection made to the database to provide a2 min read
  31. Shell Script - Check Area UtilizationScript : Progress database area utilization Script Name: progressarearpt.sh Script Function: Get the Actual DB Size,get Max area size & current utilization per2 min read
  32. Shell Script - Database StopScript : Progress Database Stop Script Name: stoppro.sh Script Function: To stop the database instance and optionally truncate bi (-t) Script Usage:1 min read
  33. Shell Script - Progress Add ExtentsScript : Progress add new extents Script Name: addextent.sh Script Function: Add new extent to the database provided in the add.st file in database path after1 min read
  34. Shell Script - Progress BackupScript : Progress database backup Script Name: progressbackup.sh Script Function: Perform a coldhot backup for a given database. Script Usage:3 min read
  35. Shell Script - Progress lk infoScript : Progress database lk file information Script Name: lkinfo.sh Script Function: Information about the database status from lk file Script Usage:1 min read
  36. Shell Script - Progress Log ParsingScript : Shell script to parse the progress database log file Script Name: dblogparse.sh Script Function: Script classifies the messages into 4 categories16 min read
  37. Shell Script - Progress Start DatabaseScript : Progress start a database instance Script Name: startpro.sh Script Function: Start a database instance of progress with a pf file. Script Usage:3 min read
  38. Shell Script - Progress Vairable Extent SizeScript : Progress check Variable Extent Size Script Name: checkextent.sh Script Function: check the size of the variable extents (< $extentMaxSize set in the1 min read
  39. Shell Script - ProgressDB Storage GrowthScript : Progress database Storage area size growth Script Name: progressgrowthrpt.sh Script Function:Calculate report of time remaining for extents to grow2 min read
  40. What is new in OE1.OpenEdge Transparent Data Encryption (TDE) By providing application transparent data encryption, TDE provides both security and performance needs using2 min read

Shell Scripting

Practical shell scripts for automation, database ops, and system administration.

8 entries
  1. Shell Script - File DetailsScript : Displaying a File with Additional Information Script Name: showfile.sh Script Function:Script as an alternative to cat displaying a files additional1 min read
  2. Shell Script - Contents of DirectoriesScript : Displaying the Contents of Directories Script Name: formatdir.sh Script Function: It generates a multi column listing of files and directories,2 min read
  3. Shell Script - Bulk Renaming FilesScript : Bulk-Renaming Files Script Name: bulkrename.sh Script Function: It takes two arguments for the text to match and replace, and a list of arguments1 min read
  4. Shell Script - Server StatisticsScript : Shell script to test the throughput of disk subsystem. Script Name: serverstat.sh Script Function: Statistics about server status,disk and memory20 min read
  5. Shell Script - Reminder UtilityScript : Reminder Utility Script Name: remember.sh, remindme.sh Script Function: The first script, remember , lets you easily save your snippets of information1 min read
  6. Shell Script - Interactive CalculatorScript : Interactive Calculator Script Name: calculator.sh Script Function:Script to run a fully interactive command line–based calculator. Script Usage:1 min read
  7. Linux ShellsUsing the Bourne Again Shell The Bourne Again Shell is usually just referred to as a Bash shell. It basically allows to enter Linux commands. In the Gnome2 min read
  8. Shell Scripting for Progress DBANow that we are familiar with the basic unix commands which would be required by a Progress DBA at various troubleshooting and day to day activity scenarios3 min read

Linux

Linux system administration, environment setup, and essential commands.

5 entries
  1. Linux InstallationTo install linux follow the below steps: To start the installation from DVD/CD set the BIOS to boot from CDROM. This installation guide is to install Linux1 min read
  2. Linux EnvironmentGnome and the Bash shell are two of the most used interfaces in Linux for managing the system. GNOME (GNU Network Object Model Environment, pronounced3 min read
  3. Basic CommandsBelow are the most commonly used linux commands and its purpose: && — and — or apropos — returns all documentation containing the defined command bzip/bunzip —5 min read
  4. Basic Unix commands for Progress DBASince the most used operating system for a progress database product is linux/unix/solaris as compared to windows OS, a Progress DBA is expected to have some7 min read
  5. Introduction to Unix/LinuxHow does the Linux OS work? Linux has a KERNEL and one or more SHELLS The shell is the command line interface through which the user interacts with the OS1 min read

Angular

Angular framework — components, routing, services, and UI integration.

7 entries
  1. Angular OverviewAngular JS components are a below:1 min read
  2. Angular Project SetupThe below steps will help to setup a new project in angular js to develop an angular js based project. 1. Check if your system has already node js by using1 min read
  3. Routing in AngularWe can route hyperlinks to route to any of the components based on url changes using single page application at client side instead of loading whole page.So1 min read
  4. Bootstrap in AngularTo implement bootstrap in an angular application after creating it with ng new command , follow the below steps: 1. Open command prompt and cd to the project1 min read
  5. Styling with Angular MaterialIn the last blog , the dashboard.component.html has been styled by using angular material by using mat card and other similar tags.Follow the below steps to1 min read
  6. Fake Service in AngularTo call a fake rest api service from our angular application , we can use a fake service using angular.Follow the below steps to create a fake server hosting a1 min read
  7. Rest Api in AngularTo call a rest api service in angular, follow the below steps after setting up a fake service and routings as mentioned in previous blogs: 1. Generate an1 min read

AWS

Amazon Web Services — cloud fundamentals and key services.

1 entries
  1. AWS IntroductionEssential Characteristics: • On Demand Self-Service • Broad network access • Resource Pooling • Rapid Elasticity • Measured Service Deployment Model • Private6 min read

General

Miscellaneous technical notes and articles.

1 entries
  1. Welcome to HelloWereldWhy I'm building a new home for my writing, projects, talks and experiments.1 min read