While reading another blog, which had some interesting use cases mentioned with some examples, one that talked about updating your static portfolio pages seemed really interesting.

I was really interested in hosting the Hashnode post in two places.

1. My GitHub profile landing page.

2. My Portfolio Website (static), which is also hosted via GitHub pages.

Since, both of them are static, and being able to update the latest few blog posts there dynamically would be cool.

So, the initial idea was simple: Write a Github action to run a script that fetches the posts from Hashnode, updates the html, and pushes to the repo. Do the same for the second one too. Too brute. So then I planned to convert them into Github Action steps instead.

Action 1: Get Latest Hashnode Posts

This GitHub Action retrieves posts from a Hashnode publication using GraphQL. The action accepts certain inputs and outputs the resulting posts in JSON format.

Inputs

• HASHNODE_PUBLICATION_ID (required): The ID of the Hashnode publication.

• HASHNODE_GQL_ENDPOINT (optional): The GraphQL endpoint for Hashnode. (default is https://gql.hashnode.com)

• MAX_POSTS (optional): Maximum number of posts to retrieve (default is 10).

Outputs

• result: JSON string containing the retrieved posts.

Example Usage

name: Update Hashnode Posts
on:
  workflow_dispatch:
  schedule:
    - cron: '0 0 * * *'

jobs:
  get_posts:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Repository
        uses: actions/checkout@v2

      - name: Get Hashnode Posts
        id: hashnode-posts
        uses: anupammajhi/githubaction-latest-hashnode-posts@v1.0.0
        with:
          HASHNODE_PUBLICATION_ID: ${{ secrets.HASHNODE_PUBLICATION_ID }}
          HASHNODE_GQL_ENDPOINT: 'https://gql.hashnode.com' # Optional
          MAX_POSTS: 8  # Optional

Action 2: Update File in Repo with EJS

This GitHub Action allows you to update a file in your repository using an EJS template.

The action is designed to replace content in a specified file between designated start and end comments with the rendered output of an EJS template. It requires five inputs, including the file path, starting comment, ending comment, EJS template path, and the JSON input for the template.

Inputs

FILE_PATH (required)

The relative path of the file in the repository that you want to update.

STARTING_COMMENT (required)

The starting comment in the file, marking the beginning of the content to be replaced.

ENDING_COMMENT (required)

The ending comment in the file, marking the end of the content to be replaced.

EJS_TEMPLATE_PATH (required)

The relative path of the EJS template file that will be used for rendering content.

TEMPLATE_INPUT_JSON (required)

The JSON input that the EJS template will use for variable extraction. Ideally, this should be obtained from another action step.

Usage

- uses: actions/checkout@v4
  with:
    # The relative path of the file in the repository that you want to update.
    FILE_PATH: ''

    # The starting comment in the file mentioned in FILE_PATH, marking the beginning of the content to be replaced.
    # This usually depends on which file you are trying to update. 
    # e.g. for html/markdown file you can use '<!-- YOUR_STARTING_COMMENT -->'
    # e.g. for js file you can use '// YOUR_STARTING_COMMENT'
    STARTING_COMMENT: ''

    # The ending comment in the file mentioned in FILE_PATH, marking the end of the content to be replaced.
    # This usually depends on which file you are trying to update. 
    # e.g. for html/markdown file you can use '<!-- YOUR_ENDING_COMMENT -->'
    # e.g. for js file you can use '// YOUR_ENDING_COMMENT'
    ENDING_COMMENT: ''

    # The relative path of the EJS template file that will be used for rendering content.
    # [Learn more about EJS templates](https://ejs.co/#docs)
    EJS_TEMPLATE_PATH: ''

    # The JSON input that the EJS template will use for variable extraction. 
    # Ideally, this should be obtained from another action step.
    TEMPLATE_INPUT_JSON: ''

Example Usage

name: Update Hashnode Posts
on:
  workflow_dispatch:
  schedule:
    - cron: '0 0 * * *'

jobs:
  get_posts:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Repository
        uses: actions/checkout@v2

      - name: Get Hashnode Posts
        id: hashnode_posts
        uses: anupammajhi/githubaction-latest-hashnode-posts@v1.0.0
        with:
          HASHNODE_PUBLICATION_ID: ${{ secrets.HASHNODE_PUBLICATION_ID }}
          HASHNODE_GQL_ENDPOINT: 'https://gql.hashnode.com' # Optional
          MAX_POSTS: 8  # Optional

      - name: Update File
        id: update_file
        uses: anupammajhi/githubaction-update-repo-file@v1.0.1
        with:
          FILE_PATH: "README.md"
          STARTING_COMMENT: "<!-- CONTENT_AUTO_START -->"
          ENDING_COMMENT: "<!-- CONTENT_AUTO_END -->"
          EJS_TEMPLATE_PATH: "readmeTemplate.ejs"
          TEMPLATE_INPUT_JSON: ${{ steps.hashnode_posts.outputs.result }}

Create EJS Template to format posts

This template is used to make sure that the data is published in our file in the consistent format that we want. EJS is a standard templating tool, and that has been used here. Here is an example.

<% 
let count = 0;
let i = 0;
while (count < 4 && i < 10){
    if(jsonData[i].coverImage){
        count++;

        let blogUrl = jsonData[i].url
        let imageUrl = jsonData[i].coverImage.url

        const replacementUrl = "tech.anupamm.com"
        const originalUrlSearchString = /anupammajhi\.hashnode\.dev/g

        blogUrl = blogUrl.replace(originalUrlSearchString, replacementUrl)
        imageUrl = imageUrl.replace(originalUrlSearchString, replacementUrl)

        let date = new Date(jsonData[i].publishedAt);
        let formattedDate = date.toLocaleString('en-US', {
            year: 'numeric',
            month: 'long',
            day: 'numeric'
        });
%>
<tr>
<td style="width:30%"><a href="<%= blogUrl %>"><img src="<%= imageUrl %>" width="500" height="auto" /></a></td>
<td>
<a href="<%= blogUrl %>"><b><%= jsonData[i].title %></b></a><br />
<code><%= formattedDate %></code><br />
<p><%= jsonData[i].brief %></p>
</td>
</a>
</tr>
<% 
    }
    i++;
} 
%>

Make sure the file that we are trying to update has the appropriate Starting and Ending Comments in place.

That's it. Now we have static pages hosted in Github that are updated daily with the posts from Hashnode.

However, this extremely helpful blog post has helped me earlier, I successfully installed RHEL8 in WSL2 about a year back. In this post, I have captured those steps, but for RHEL9, so that I can come back to this again when I need it. Along the way, hopefully, this also helps someone looking to do the same.

Prerequisites:

Before diving into the installation process, make sure you have the following prerequisites:

1. OS: Windows 10 Pro or above.

2. Enable WSL2: Ensure that WSL2 is enabled on your Windows machine. You can follow Microsoft's official documentation for guidance.

3. Docker: To get the Red Hat image from the Red Hat registry.

4. Terminal [Optional]: Windows Terminal or even Powershell terminal would work.

5. Red Hat account: Create a free developer account to be able to connect to the Red Hat registry.

6. Basic knowledge of creating and editing files in Linux.

Prepare RHEL9 Image

The most straightforward method for establishing a new WSL distro involves importing a container export file. To access Red Hat Enterprise Linux (RHEL) container images, it is necessary to retrieve them from Red Hat's dedicated registry.

Interestingly, a search for RHEL base images yields only RHEL6 and RHEL7 fully featured images. However, when exploring build images, an RHEL9 image is available, such as the one associated with the Go Toolkit.

Let's proceed to create a container using the Go RHEL9 image. It's essential to note that we won't execute any commands within the container, causing it to be created and exit immediately. This behavior aligns with our intention, as we plan to export the container to a file.

Open your Powershell Terminal and login to the Red Hat registry with the username and password for Red Hat developer account. Then we will get the image from the registry.

# Login to the Red Hat registry
# Enter the Red Hat account username and password
docker login registry.redhat.io

# Pull the GO image with the RHEL9 base image
docker run --name rhel9 registry.redhat.io/rhel9/go-toolset

Once we create the container, we can now export it as an archive file.

I am using PowerShell. If you are using another distro within wsl2, then you can use the location as /mnt/c/wslsources instead

# Create directory to save the archived image
mkdir C:\wslsources

# Export the container to an archive file
# Command help:
# - base command: docker export
# - option to output to a file: -o C:\wslsources\rhel9.tar.gz
# - name of the container to be exported: rhel9
docker export -o C:\wslsources\rhel9.tar.gz rhel9

# Check if the export has been successful
ls -l C:\wslsources

Import RHEL9 to WSL

We can now use this archived image to import it to WSL as a new WSL2 distro.

# Create a new directory for containing the WSL custom distros, or move into it if already created
mkdir c:\wsldistros
cd c:\wsldistros

# Import the container file as a WSL v2 distro
# Command help:
# - base command: wsl --import
# - name of the distro: rhel9
# - location of the distro files: ./rhel9
# - path of the container file: C:\wslsources\rhel9.tar.gz
wsl --import rhel9 ./rhel9 C:\wslsources\rhel9.tar.gz

# Launch the new distro
wsl -d rhel9

Note: the command above assumes that WSL is set to have the v2 versions by default. If there’s an uncertainty about which version is actually setup, the following option can be added to the command: --version 2

Check the OS release version to be sure that we are in the right distro.

# Check with version is installed
cat /etc/os-release

Finalizing RHEL9 setup

Once we have RHEL, it’s time to start the configuration. But first, let’s register it.

# Register the system with our Red Hat username and password
subscription-manager register

# Set a role to the System
# It can be one suggested by Red Hat, or in our case, let's have "some fun", and set it to WSL
subscription-manager role --set="Red Hat Enterprise Linux WSL"

# Set a support level
# In our case, remember we have a free Developer license and the only valid support is "Self-Support"
subscription-manager service-level --set="Self-Support"

# Set a usage (read: purpose) to the System
# We will pick "Development/Test" for this first try
subscription-manager usage --set="Development/Test"

# Once everything is set, we can attach the system to our Red Hat subscription
subscription-manager attach

Note: If Simple Content Access (SCA) is enabled in your Red Hat developer account, then the 'attach' would be ignored, just like in my case, and that is okay.

[Optional] We can confirm if the system has been correctly subscribed in the Red Hat Customer Portal by clicking in our Subscription and picking the “Systems” tab

Now we can go ahead and update the packages and install additional packages to make is a full fledged RHEL9 setup.

# Update the system
dnf update

# Add sudo, zsh and cracklib-dicts
dnf install -y sudo zsh cracklib-dicts

Note: while sudo and zsh are familiar applications among Linux users, cracklib-dicts is a package utilized by the passwd command for password resets. The container image does not inherently include this package. Including cracklib-dicts in the container will prevent the occurrence of a warning message each time a password is changed for any user.

With packages installed, we can create our main user.

# Create a new user with the desired username
# Command help:
# base command: useradd
# create the home directory: -m
# set ZSH as the default shell: -s /bin/zsh
# add the user to the group 'wheel' which is defined in the 'sudoers' file: -G wheel
# username desired: majhi
useradd -m -s /bin/zsh -G wheel majhi

# Set a password for the user created
passwd majhi

# Set a password for the 'root' user
passwd

In the final phase of configuration, attention is directed towards WSL itself. We will craft a configuration file specifically for the 'interoperability' of our newly customized distribution.

This configuration file resides 'on the WSL side', implying that its effects are limited to the distro it is associated with"

# Create the configuration file with a preferred text editor
vi /etc/wsl.conf

## This is the file content
## Replace the username and hostname as desired
## Source for the options: https://docs.microsoft.com/en-us/windows/wsl/wsl-config#configure-per-distro-launch-settings-with-wslconf
[automount]
enabled = true
mountfstab = true
root = /mnt/
options = metadata,uid=1000,gid=1000,umask=0022,fmask=11,case=off

[network]
generatehosts = true
generateresolvconf = true
hostname = rhel9

[interop]
enabled = true
appendwindowspath = true

[user]
default = majhi

# [Optional] Check if the file has been correctly created
cat /etc/wsl.conf

Now open a new PowerShell terminal to reload the rhel9 from wsl, for the settings to take effect.

# Stop the custom distro only from PowerShell 
wsl --terminate rhel9

# Start the distro
wsl -d rhel9

Now it is logged in as the new main user that we created.

After this, we can also go ahead and do some cosmetic changes to our zsh shell to make it look pretty and add some theme. I have covered that in another post, the "zsh" Theme I use

SystemD setup

When examining the disparities between WSL distros and fully-fledged Linux distributions, one notable absence is the lack of a service manager. This omission is a deliberate result of the customized init process that facilitates broad interoperability.

A new option in the WSL configuration file now enables the execution of a command at the distro's boot time. This modification introduces a significantly cleaner and more efficient method for initiating SystemD.

# Edit the wsl configuration file with a preferred text editor
sudo vi /etc/wsl.conf

## Add the new boot section with the command to start SystemD at the end of the file
## Source: https://github.com/diddlesnaps/one-script-wsl2-systemd/tree/build-21286%2B
[boot]
command = "/usr/bin/env -i /usr/bin/unshare --fork --mount-proc --pid -- sh -c 'mount -t binfmt_misc binfmt_misc /proc/sys/fs/binfmt_misc; exec /usr/lib/systemd/systemd --unit=multi-user.target'"

# [Optional] Check if the change has been successfully saved
cat /etc/wsl.conf

Then we will need to create a script to be launched with the distro starts:

# Create the script to be launched when the distro starts
sudo vi /etc/profile.d/00-wsl2-systemd.sh

## The file should have the following content
if [ "$(ps -c -p 1 -o command=)" != "systemd" ]; then
        if [ -z "$SUDO_USER" ]; then
                export > "$HOME/.profile-systemd"
        fi

        if [ "$USER" != "root" ]; then
                exec sudo /bin/sh "/etc/profile.d/00-wsl2-systemd.sh"
        fi

        if ! grep -q WSL_INTEROP /etc/environment; then
                echo "WSL_INTEROP='/run/WSL/$(ls -r /run/WSL | head -n1)'" >> /etc/environment
        fi

        if ! grep -q DISPLAY /etc/environment; then
                echo "DISPLAY='$(awk '/nameserver/ { print $2":0" }' /etc/resolv.conf)'" >> /etc/environment
        fi

        exec /usr/bin/nsenter --mount --pid --target "$(ps -C systemd -o pid= | head -n1)" -- su - "$SUDO_USER"
fi

if [ -f "$HOME/.profile-systemd" ]; then
        source "$HOME/.profile-systemd"
fi

if [ -d "$HOME/.wslprofile.d" ]; then
        for script in "$HOME/.wslprofile.d/"*; do
                source "$script"
        done
        unset script
fi

# [Optional] Check if the file has been correctly saved
cat /etc/profile.d/00-wsl2-systemd.sh

Finally, we will need to update the sudoers file so that this can be launched without using a password every time the distro starts.

# Edit the sudoers file
sudo visudo

## Add a new value to the environment variables that are kept
Defaults env_keep += "WSL_INTEROP"

## Add the command to run the script without password
%wheel ALL=(root) NOPASSWD: /bin/sh /etc/profile.d/00-wsl2-systemd.sh

# [Optional] Check if the sudoers have been correctly updated
sudo grep -i wsl /etc/sudoers

Then we just reload the distro for the settings to take effect.

# Stop the custom distro only from PowerShell 
wsl --terminate rhel9

# Start the distro
wsl -d rhel9

To confirm after launching a new session of our distro and check SystemD processes are running, we can check for PID 1:

ps -ef

Setup Docker

This RHEL9 setup still doesn't have any docker client or daemon. However, we can use the following steps as per Docker documentation.

# Install the yum-utils package
sudo yum install -y yum-utils

# Add the Docker repository
sudo yum-config-manager \
    --add-repo \
    https://download.docker.com/linux/centos/docker-ce.repo

# Install Docker
sudo yum install docker-ce docker-ce-cli containerd.io

## We will need approve the import of the Docker GPG key
## Just type 'y' and press Enter

Before initiating Docker, a necessary modification is required in the Docker SystemD file. The challenge arises from the fact that Red Hat no longer utilizes iptables but has transitioned to its successor, nftables.

Fortunately, we can address this by adjusting the Docker start command to eliminate the reliance on searching for iptables.

# Update the Docker SystemD service
sudo vi /usr/lib/systemd/system/docker.service

## Locate the ExecStart option and add --iptables=false at the end of the line
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --iptables=false

# [Optional] Check if the file has been correctly saved
grep ExecStart /usr/lib/systemd/system/docker.service

# Reload the SystemD service configuration files
sudo systemctl daemon-reload

# Start Docker
sudo systemctl start docker

# Check if Docker has been correctly started
sudo systemctl status docker

# Enable Docker to be started when the distro starts
sudo systemctl enable docker

In order for us to use Docker without the sudo command, we need to add our user to the docker group

# Add our user to the docker group
sudo usermod -aG docker $USER

# Refresh the group docker without launching a new shell
newgrp docker

# Check the Docker version
docker version

That's it! We now have a full-blown RHEL 9 distro running in WSL with necessary WSL interoperability settings.

What an enjoyable journey, and if there's one key takeaway from this blog, it's that WSL2 serves as a robust platform for running Linux. Thanks to

some clever scripting, the experience is steadily approaching that of a virtual machine (VM) or bare-metal setup.

It's crucial to note, however, that achieving official support might not always be feasible, introducing potential limitations and roadblocks. Nevertheless, from a technical standpoint, it was entirely achievable.

About CODEOWNERS?

GitHub released the code owners functionality somewhere around 2017, which allows you to designate who owns the code inside a repository. Teams can use this to assign people or groups to manage particular sections of the codebase. A file called CODEOWNERS, which is found at the repository's root, lists the code owners.

How Does it Work?

Patterns in the CODEOWNERS file correspond to file paths in your repository. Every pattern has one or more GitHub usernames or team names attached to it, which identify the people or groups in charge of checking and updating the code inside that path.

For example, a CODEOWNERS file might look like this:

# Assigning code ownership

*.js @frontend-team

backend/* @backend-team

/docs/* @documentation-team

In this example, files in the "backend" directory belong to the backend team, files in the "docs" directory to the documentation team, and JavaScript files to the frontend team.

Benefits

1. Clear Ownership and Responsibility

Makes it explicit who is responsible for reviewing and maintaining specific parts of the codebase. This ensures that changes are overseen by those with the most knowledge of and responsibility for the affected code.

2. Granular Code Review Process

The code review procedure runs smoothly when there are assigned owners for every piece of the codebase. Pull requests streamline the review process and lessen the possibility of changes being missed by automatically assigning reviewers based on the CODEOWNERS file.

3. Improved Collaboration

This encourages team members to communicate with one another. Developers get to know who to contact for advice or clarification on particular code regions.

4. Reduced Bottlenecks

By distributing ownership, it helps prevent bottlenecks caused by a single individual or team being overloaded with code review requests. This promotes a more balanced and scalable development process.

Implementing CODEOWNERS in Your Project

1. Create a CODEOWNERS file:

• Create a CODEOWNERS file at the root of your repository.

• Specify patterns and assign owners based on your project's structure.

2. Review and Update Regularly:

• Regularly review and update the CODEOWNERS file to reflect changes in team structure or project organization.

3. Integrate with Workflows:

• Ensure that your team integrates it into their workflows. Utilize features like automatic pull request assignment based on code ownership.

4. Documentation:

• Document your Code Owners conventions and processes to facilitate onboarding for new team members.

There you go. Consider implementing CODEOWNERS in your next project to manage code reviewers at the directory/file level and elevate your team's collaboration and code management practices.

Microsoft provides a not so beautiful but powerful command to do that too. Drumrolls......

The Runas Command

The `runas` command in Windows allows users to run a program with different user credentials than the ones currently in use. To open PowerShell in administrator mode with a domain admin user, the following command can be used:

runas /user:DOMAIN\adminuser "powershell.exe -NoProfile -ExecutionPolicy Bypass -Command \"Start-Process powershell -ArgumentList '-NoProfile -NoExit' -Verb RunAs\""

NOTE: Remember to replace placeholders like "DOMAIN" and "adminuser" with your actual domain name and admin username.

This command will give a command line prompt to enter password.

This command utilizes the `-NoExit` parameter, ensuring that the PowerShell console remains open after execution.

Breaking Down the Command

- runas /user:DOMAIN\adminuser: Specifies the domain and admin user for which elevated privileges are required.

- powershell.exe -NoProfile -ExecutionPolicy Bypass: Launches PowerShell with specific settings. The `-NoProfile` flag ensures a minimal profile is used, and `-ExecutionPolicy Bypass` allows the execution of scripts.

- Start-Process powershell: Initiates a new PowerShell process.

- -ArgumentList '-NoProfile -NoExit': Passes additional arguments to the new PowerShell process. The `-NoExit` parameter prevents the console from closing immediately.

- -Verb RunAs: Requests to run the process with elevated permissions.

Use Cases

- Script Execution: Running scripts that require administrative privileges.

- System Configurations: Making changes to system settings that necessitate elevated access.

- Software Installations: Installing software that requires administrative rights.

While doing the setup, I found this zsh theme very nice, and I have been using it since then. The above mentioned blog already has the steps, but it has a lot of other info about setting up RHEL in WSL2 as well. So here is the extract of setting up the zsh theme (steps are for RHEL8, might differ for other Linux dists).

1. Setup ZShell

If the zsh is not already installed, then install zsh.

dnf install -y sudo zsh cracklib-dicts

2. Install "Oh My Zsh"

Oh My Zsh is the configuration manager being used for this

sh -c "$(curl -fsSL https://raw.githubusercontent.com/ohmyzsh/ohmyzsh/master/tools/install.sh)"

3. Update user to use zsh as the default shell

Find the location of zsh installation

cat /etc/shells

Use vi editor to update the default shell of the user to zsh by replacing the path (example: ec2-user is the user we are working with)

sudo vi /etc/passwd

4. Update zsh config file

As written in the installation message, we can now change the zsh config file. For now, we will only change the theme to Fino Theme.

vi $HOME/.zshrc

In the opened vi editor, look for the key ZSH_THEME and update it as below:

ZSH_THEME="fino-time"

Check if the config file got updated, and then load the new config file.

# [Optional] Check if the theme value has been correctly saved
grep ZSH_THEME $HOME/.zshrc

# Reload the zsh configuration file
source $HOME/.zshrc

5. Update theme settings

Open the theme settings in vi editor

vi $HOME/.oh-my-zsh/themes/fino-time.zsh-theme

Look for the PROMPT variable and modify it to add the variables

PROMPT="╭─%{$FG[040]%}%n%{$reset_color%} %{$FG[239]%}at%{$reset_color%} %{$FG[033]%}$(box_name) (WSL: $WSL_DISTRO_NAME | kernel: $(uname -r))%{$reset_color%} %{$FG[239]%}in%{$reset_color%} %{$terminfo[bold]$FG[226]%}%~%{$reset_color%}\$(git_prompt_info)\$(ruby_prompt_info)
╰─ %* \$(virtualenv_info)\$(prompt_char) "

Reload the zsh configuration file, it will also reload the oh my zsh theme

source $HOME/.zshrc

That's it, now we have a nice-looking theme that gives a lot more information without being too overwhelming. It has a nice clock in the prompt too.

You also might have already figured that the easiest way to do this is to change the host name of the Windows host on which you are running WSL. That’s probably obvious and not something you are looking for.

So, if you are trying to change the hostname of the Ubuntu distribution running in WSL without changing the host name of the Windows host system, there are few steps to follow.

Step 1: Note down your ubuntu hostname

Make sure you note down your current ubuntu hostname, as we might need this going forward.

amajhi@ANUPAM-DESKTOP:~$ hostname
ANUPAM-DESKTOP

In this example ANUPAM-DESKTOP is the hostname which we will be replacing with the new hostname KATANA

Step 2: Update the wsl.conf file

Next you will need to update (or create if not present) the file /etc/wsl.conf

amajhi@ANUPAM-DESKTOP:~$ sudo vi /etc/wsl.conf
[sudo] password for amajhi:

Then add/update the following lines to the file.

[network]
hostname = KATANA
generateHosts = false

The “generateHosts = false” makes sure that he hostname and hosts files are not replaced back to original on reboot.

Step 3: Update the hosts file

This is an important step to prevent any host resolution issue. If this step is not done, then the system might get confused with what the hostname really is!

amajhi@KATANA:~$ sudo apt update
sudo: unable to resolve host KATANA: Name or service not known

Let’s look at the current hosts file and make it right.

amajhi@KATANA:~$ sudo vi /etc/hosts

# This file was automatically generated by WSL. To stop automatic generation of this file, add the following entry to /etc/wsl.conf:
# [network]
# generateHosts = false
127.0.0.1 localhost
127.0.1.1 ANUPAM-DESKTOP.localdomain ANUPAM-DESKTOP

# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

To do the appropriate changes we need to update the /etc/hosts file and replace all occurrences of the old hostname ANUPAM-DESKTOP with the new hostname KATANA.

# This file was automatically generated by WSL. To stop automatic generation of this file, add the following entry to /etc/wsl.conf:
# [network]
# generateHosts = false
127.0.0.1 localhost
127.0.1.1 KATANA.localdomain KATANA

# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

Step 4: Restart the WSL instance

To restart the WSL instance we need to shut down the WSL instance and relaunch ubuntu.

PS C:\Users\amajhi> wsl --shutdown

or

PS C:\Users\amajhi> wsl -t Ubuntu

Once you launch the ubuntu in WSL again, you should see the updated hostname intact and should be able to resolve host too. Let’s verify the resolution.

amajhi@KATANA:~$ sudo apt update
[sudo] password for amajhi:
Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease
Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease
Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease
Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
64 packages can be upgraded. Run 'apt list --upgradable' to see them.

That’s it!

KNN around us

Though this is over-simplified, you would have noticed around yourself that people with similar interests usually have similar traits. Let’s think of a few examples:

People following similar kinds of news headlines and in a similar age group can roughly be classified into the political parties they will most likely vote for. People with similar calorie intake and a similar amount of physical activity can be classified as healthy or not healthy. Emails with similar choices of words and similar formats can be classified as spam or not spam and so on …

How does KNN work?

The letter k in a kNN is the number of nearest neighbors that it will take into account to decide to which class an object belongs. So, quite simply, if k=10, we check the 10 closest neighbors of the object we want to classify, and if it has more neighbors of type A and fewer neighbors of other types B or C, we say the object belongs to type/class A.

But how do we measure which is closest? The KNN algorithm compares the position of the data point in question with all the training data points (Check out this blog if you are not sure what training a model is). To find the distance we can use any distance metric, even Euclidian distance. Once the model finds out who are its ‘k’ closest neighbors, it just goes ahead with voting among those members to find out where it belongs.

Model trying to find where the yellow star belongs based on 5 nearest neighbors

In the above figure, the model is trying to figure out which category the uncategorized data point (yellow star) belongs to (assuming k=5). We can easily see here that out of 5 nearest neighbors, 4 belong to the red squares category. Hence, the yellow star should also be a red square category data point.

What is the best value of ‘k’?

Firstly, there is no single best universal value of k. Then how do we decide what value of k to use? The answer is it depends. It depends on the data, more precisely the training and validation data. While training and validating with different k values we have to make sure we get the least error possible, and then it’s up to us to decide which ‘k’ value to use.

Example: Validation error for different values of k image source: AnalyticsVidya

In the example image above, the plot between different k values vs. validation error shows that we get the lowest error value somewhere around k=10. Hence, in this case, k=10 should be our choice.

KNN and Curse of Dimensionality

Just like any other data science algorithm, KNN faces the curse of dimensionality. The curse of Dimensionality is basically the difficulty/error that a model faces as we keep increasing the number of features/dimensions. For example, a model using a few features (e.g. age, sex, height) would most likely predict results better than a model using a huge number of such features.

To understand better, let’s imagine we are trying to find out the shoe size of a person based on age and height. Here we just have 2 dimensions to check, i.e. age and height, hence there can only be certain kinds of people with a certain amount of differences/individuality/noise. So we can quite accurately predict what the shoe size could be. But what if the data also has additional dimensions like weight, city, and favorite color? The model can still predict well enough, but there are high chances that the prediction wouldn’t be accurate enough. The reason is that two people may live in different cities but still have similar physical features, but in KNN, those two people will not be very close neighbors. Consider ‘favorite color’ as a feature, that will make it even more difficult.

The solution is to remove unnecessary or irrelevant features/dimensions and keep the number of features to a minimum.

KNN — Regression

I did mention in the beginning that KNN is a classification algorithm. However, KNN is also used for regression models. For the unaware, a regression model is where we try to find out the actual value instead of a category, e.g. trying to predict height, salary, price, etc. The only difference in KNN for Regression compared to KNN for classification is that, instead of voting among closest neighbors, the model takes an average of the actual value (e.g. salary) of all the k nearest neighbors. There can be different ways to calculate the average.

That brings us to the end of this post on KNN introduction.

All this started back in 2003 when Google released its paper on Google File System. Later the Hadoop project started taking that as a blueprint and it has reached here today.

Hadoop 1.x had quite a different structure and had certain limitations, which have been addressed in Hadoop 2.x versions with a different architecture with different components to enable more reliable, distributed, parallel processing of Big Data.

Hadoop comprises of 2 main core components:

1. HDFS (Hadoop Distributed File System) — Storage Unit
2. YARN (Yet Another Resource Negotiator) — Processing Unit

HDFS

HDFS comprises of 3 components.

Name Node

The name Node is the ‘master’ component of HDFS system. It stores the metadata of HDFS, which is essentially the directory tree of the files in HDFS, and a track of all the files across the cluster.

The name node does not store the actual data. It knows the list of blocks and locations for any given file.

When the name node is down, the cluster itself is considered down as the cluster becomes inaccessible.

It stores fsimage(snapshot of the filesystem at start-up) and edit logs (sequence of changes made to the filesystem after start-up).

Secondary Name Node

The secondary Name node is NOT a backup name node as the name might suggest. It is more of an assistive node.

It is responsible for taking checkpoints of the file system metadata present in the name node. It does this by checkpointing fsimage. It gets the edit logs from the name node in regular intervals and creates the checkpoint. Then it copies this fsimage back to the name node.

The name node uses this fsimage in the next start-up so that it reduces start-up time.

Data Node

This is where the actual data is stored. The name node keeps track of each block that is stored in these data nodes. It is also known as the ‘slave’.

When it starts up, it announces its presence to name node and also the data blocks it is responsible for. This is the workhorse of Hadoop HDFS.

YARN

YARN comprises of 2 components:

Resource Manager

The resource Manager is responsible for taking inventory of available resources and running critical services, the most critical of which is the Scheduler.

The scheduler allocates resources. It negotiates the available resources in the cluster and manages the distributed processing. It works along with the Node Manager to attain this.

Node Manager

The node manager acts as the ‘slave’ to the Resource Manager.

It keeps track of the tasks and jobs that are being deployed to the data nodes. It helps the Resource manager keep track of available space and processing power. Memory, bandwidth, etc. so that tasks can be distributed to the data nodes.

ENTRYPOINT vs CMD:

ENTRYPOINT:

The ENTRYPOINT instruction sets the primary command that will executed by the container when it starts. It can be specified with or without an executable and also allows for arguments.

ENTRYPOINT ["nginx", "-g", "daemon off;"]

ENTRYPOINT command cannot be overridden by appending a command while executing 'docker run' at runtime. However, the arguments can be appended.

Example Dockerfile:

FROM ubuntu
ENTRYPOINT ["echo", "Hello"]

Run the container with a command:

docker run my-image Goodbye

Output:

# output
Hello Goodbye

If we really want to overwrite the command in ENTRYPOINT, we need to use the '--entrypoint' with docker run:

# Example
docker run -it --entrypoint /bin/sh my_custom_image

CMD:

The CMD instruction sets the default command for the container. If used without an ENTRYPOINT, it becomes the default command and arguments that will be used when the container starts. Whereas, when used with an ENTRYPOINT, it provides default arguments for the entry point.

This can be overridden though at runtime by passing a command when running the container using 'docker run'.

Example:

CMD ["nginx", "-g", "daemon off;"]

CMD command can be overridden by appending a command while executing 'docker run' at runtime.

Example Dockerfile:

FROM ubuntu
CMD ["echo", "Hello"]

Run the container with a command:

docker run my-image Goodbye

Output:

# output
Goodbye

Best Practices:

We should prefer the exec form of the ENTRYPOINT and CMD to avoid shell processing.

Example:

ENTRYPOINT ["echo", "Hello, $USER"]

If we use the insecure way of using shell form instead, it might introduce a security vulnerability. The shell form can be abused to perform shell injection. Additionally, the shell form becomes dependent on the particular shell in use and if a different shell is being used in a distribution, it might not be consistent.

# An example where $USER can be used for shell injection vulnerability.
# Malicious code can be introduced if the $USER variable is not sanitized.
FROM ubuntu
CMD echo "Hello, $USER"

Choosing Between CMD and ENTRYPOINT

• Use CMD when you want to provide default values for the command and allow them to be easily overridden at runtime.

• Use ENTRYPOINT when you want to set a fixed primary command and allow users to append additional arguments.

In many cases, combining both instructions offers a balance of flexibility and definition. Understanding these use cases will empower you to create more versatile and user-friendly Docker images.

ENTRYPOINT as a Script:

If we have a scenario where we want to run a script instead of any particular command, and maybe pass a default argument, we can do it seamlessly with ENTRYPOINT and using CMD along with it.

FROM node

# Set the working directory
WORKDIR /app

# Copy the application files
COPY . .

# Set the script as the ENTRYPOINT
ENTRYPOINT ["./entrypoint.sh"]

# Default argument 'start' to pass to the script if nothing mentioned in run.
CMD ["start"]

Conclusion:

Mastering Dockerfile entry points empowers you to create efficient, modular, and customizable Docker images. Whether you're building a simple web server or a complex microservices architecture, understanding how to leverage ENTRYPOINT and CMD is essential for Dockerfile success.

Happy Dockerizing!

Since there are not many options or tools available in Orchestrator which could help us troubleshoot or analyze the Orchestrator environment, few SQL queries come in handy. It will also be useful if you are trying to generate some kind of report of the environment.

Here are some of those useful queries.

Get Runbook (with date) and Folder Path

This will list all the runbooks Orchestrator along with its location path (folder structure). This also gives you the ‘Last Modified’ date and time for each runbook, just in case it’s important for you to filter out older runbooks.

use orchestrator;

with RunbookPath as
(
select 'Policies\' + cast(name as varchar(max)) as [path], uniqueid from dbo.folders b
where b.ParentID='00000000-0000-0000-0000-000000000000' and disabled = 0 and deleted= 0
union all
select cast(c.[path] + '\' + cast(b.name as varchar(max)) as varchar(max)), b.uniqueid from dbo.FOLDERS b
inner join
RunbookPath c on b.ParentID = c.UniqueID
where b.Disabled = 0 and b.Deleted = 0
)

SELECT f.path AS RunbookPath,r.name AS RunbookName, r.LastModified AS 'Runbook Last Modified' from POLICIES as r 
INNER JOIN RunbookPath as f on r.ParentID = f.UniqueID
WHERE r.Deleted = 0

Get Running Runbooks

This query will help you fetch all the runbooks (along with path as previous) that’s running and has not yet completed. You may use it to understand if any runbook is running that actually should not be running. There is indeed a tool available to do this already, but it does not provide the path of runbook, in such a situation if we have multiple runbooks with same name, it becomes difficult to search.

use orchestrator;

with RunbookPath as
(
select 'Policies\' + cast(name as varchar(max)) as [path], uniqueid from dbo.folders b
where b.ParentID='00000000-0000-0000-0000-000000000000' and disabled = 0 and deleted= 0
union all
select cast(c.[path] + '\' + cast(b.name as varchar(max)) as varchar(max)), b.uniqueid from dbo.FOLDERS b
inner join
RunbookPath c on b.ParentID = c.UniqueID
where b.Disabled = 0 and b.Deleted = 0
)

select f.path AS RunbookPath,r.name AS RunbookName,A.[Computer] as RunbookServer from POLICIES as r 
inner join RunbookPath as f on r.ParentID = f.UniqueID
inner join [Microsoft.SystemCenter.Orchestrator.Runtime.Internal].[Jobs] as J on J.RunbookId = r.UniqueID
INNER JOIN [dbo].[ACTIONSERVERS] A ON J.RunbookServerId = A.UniqueID
WHERE r.Deleted = 0 AND
J.[StatusId] = 1

Get Runbooks with ‘Common-Data Log’ and ‘Activity-Specific Log’ Enabled

Ever wondered how to find which Runbooks have the ‘Common-Data Logging’ and ‘Activity-Specific Logging’ enabled under runbook properties?

The following query will help you find that out. Here ‘1’ means enabled/checked and ‘0’ means disabled/unchecked

use orchestrator;

with RunbookPath as
(
select 'Policies\' + cast(name as varchar(max)) as [path], uniqueid from dbo.folders b
where b.ParentID='00000000-0000-0000-0000-000000000000' and disabled = 0 and deleted= 0
union all
select cast(c.[path] + '\' + cast(b.name as varchar(max)) as varchar(max)), b.uniqueid from dbo.FOLDERS b
inner join
RunbookPath c on b.ParentID = c.UniqueID
where b.Disabled = 0 and b.Deleted = 0
)

select f.path AS RunbookPath,r.name AS RunbookName,r.LogCommonData, r.LogSpecificData from POLICIES as r 
inner join
RunbookPath as f on r.ParentID = f.UniqueID
where r.Deleted = 0 AND (r.LogCommonData = 1 OR r.LogSpecificData = 1)

Get All ‘Powershell / C# / VB.Net’ Scripts

This query will list down all the scripts that use the inbuilt activity ‘Run .Net Script’.

You may use generate a report of all the scripts. In my work, I have used this query (with small modification) to list down all scripts that use a particular password hard-coded, so that I could fix those to avoid security risk.

Note: This query might take a long time to execute depending on how many scripts you have in the Orchestrator.

use orchestrator;

with RunbookPath as
(
select 'Policies\' + cast(name as varchar(max)) as [path], uniqueid from dbo.folders b
where b.ParentID='00000000-0000-0000-0000-000000000000' and disabled = 0 and deleted= 0
union all
select cast(c.[path] + '\' + cast(b.name as varchar(max)) as varchar(max)), b.uniqueid from dbo.FOLDERS b
inner join
RunbookPath c on b.ParentID = c.UniqueID
where b.Disabled = 0 and b.Deleted = 0
)

SELECT F.Path as RunbookPath,RB.Name AS RunBook,ACT.Name AS Activity,PS.ScriptType,PS.ScriptBody,PS.PublishedData,Act.Enabled as ActivityEnabled,RB.LastModified  FROM [Orchestrator].[dbo].[RUNDOTNETSCRIPT] AS PS
INNER JOIN [Orchestrator].[dbo].[OBJECTS] AS Act ON PS.UniqueID = Act.UniqueID
INNER JOIN [Orchestrator].[dbo].[POLICIES] AS RB ON RB.UniqueID = Act.ParentID 
INNER JOIN RunbookPath AS F ON F.UniqueID = RB.ParentID
where PS.Scriptbody is not Null
AND Act.Deleted = 0 AND RB.Deleted = 0
ORDER BY RB.LastModified,F.Path

Get All Activities Objects

If you want to list down all the runbook activities, this query will be very useful.

Pro Tip: You can use the ‘ObjectType’ column to filter the activities by type of activity.

use orchestrator;

with RunbookPath as
(
select 'Policies\' + cast(name as varchar(max)) as [path], uniqueid from dbo.folders b
where b.ParentID='00000000-0000-0000-0000-000000000000' and disabled = 0 and deleted= 0
union all
select cast(c.[path] + '\' + cast(b.name as varchar(max)) as varchar(max)), b.uniqueid from dbo.FOLDERS b
inner join
RunbookPath c on b.ParentID = c.UniqueID
where b.Disabled = 0 and b.Deleted = 0
)


/** 7A65BD17-9532-4D07-A6DA-E0F89FA0203E is the code for Link Objects, Remove the condition (AND Act.ObjectType <> '7A65BD17-9532-4D07-A6DA-E0F89FA0203E') if you want LINK objects too **/

SELECT F.Path,RB.Name AS RunBook,ACT.Name AS Activity, RB.LastModified AS 'Runbook Last Modified'  
FROM [Orchestrator].[dbo].[OBJECTS] AS Act
INNER JOIN [Orchestrator].[dbo].[POLICIES] AS RB ON RB.UniqueID = Act.ParentID 
INNER JOIN RunbookPath AS F ON F.UniqueID = RB.ParentID
where Act.Deleted = 0 AND rb.Deleted = 0 AND Act.ObjectType <> '7A65BD17-9532-4D07-A6DA-E0F89FA0203E'
ORDER BY F.Path, RB.Name , Act.Name

Get Checked Out Runbooks

Sometimes we check-out the runbooks to edit or maybe for some other reason and forget to check-in; Or maybe someone else has checked-out to work on it. And when we run the runbook it either fails or uses the older version of the runbook.

So, it becomes essential to find out if there is any runbook that has been checked out and who has checked it out. The following query might be useful for you in such scenarios.

use orchestrator;

with RunbookPath as
(
select 'Policies\' + cast(name as varchar(max)) as [path], uniqueid from dbo.folders b
where b.ParentID='00000000-0000-0000-0000-000000000000' and disabled = 0 and deleted= 0
union all
select cast(c.[path] + '\' + cast(b.name as varchar(max)) as varchar(max)), b.uniqueid from dbo.FOLDERS b
inner join
RunbookPath c on b.ParentID = c.UniqueID
where b.Disabled = 0 and b.Deleted = 0
)


/** 7A65BD17-9532-4D07-A6DA-E0F89FA0203E is the code for Link Objects, Remove the condition (AND Act.ObjectType <> '7A65BD17-9532-4D07-A6DA-E0F89FA0203E') if you want LINK objects too **/

SELECT F.Path,RB.Name AS RunBook,ACT.Name AS Activity, RB.LastModified AS 'Runbook Last Modified'  
FROM [Orchestrator].[dbo].[OBJECTS] AS Act
INNER JOIN [Orchestrator].[dbo].[POLICIES] AS RB ON RB.UniqueID = Act.ParentID 
INNER JOIN RunbookPath AS F ON F.UniqueID = RB.ParentID
where Act.Deleted = 0 AND rb.Deleted = 0 AND Act.ObjectType <> '7A65BD17-9532-4D07-A6DA-E0F89FA0203E'
ORDER BY F.Path, RB.Name , Act.Name

Get SQL Query Passwords (Encrypted)

This query will get the ‘Query Database’ activities along with the encrypted passwords that is used in the activity.

Note that the password is encrypted and has a specific 3-part (actually 4-part) format. I will leave that to you to analyze. You can use the queries coming-up later in this article to even de-crypt those passwords.

use orchestrator;

with RunbookPath as
(
select 'Policies\' + cast(name as varchar(max)) as [path], uniqueid from dbo.folders b
where b.ParentID='00000000-0000-0000-0000-000000000000' and disabled = 0 and deleted= 0
union all
select cast(c.[path] + '\' + cast(b.name as varchar(max)) as varchar(max)), b.uniqueid from dbo.FOLDERS b
inner join
RunbookPath c on b.ParentID = c.UniqueID
where b.Disabled = 0 and b.Deleted = 0
)

select f.path,r.name,Act.Name as Activity, s.password from POLICIES as r 
inner join RunbookPath as f on r.ParentID = f.UniqueID
INNER JOIN [Orchestrator].[dbo].[OBJECTS] AS Act ON Act.ParentID = r.UniqueID
INNER JOIN [Orchestrator].[dbo].[TASK_QUERYDATABASE] as S on s.UniqueID = Act.UniqueID
where r.Deleted = 0 AND Act.Deleted = 0
-- AND s.Password <> '\`d.T.~Ec/\`d.T.~De/\`d.T.~De/' /** Use this condition if you want filter-out the ones that use system password **/

Decrypt Single Orchestrator Encrypted Value

In case you come around an encrypted variable or value that you want to decrypt and find out, here is the query for that.

use orchestrator;

DECLARE @EncryptedPass nvarchar(255);
/** Set the value of @EncryptedPass as the encrypted value that you want to decrypt. Note that it is the Alphanumeric part of the Encrypted text. **/
SET @EncryptedPass = '00224EAFD0B0FC41DA40650F2F11F68801000000B60FCDF1618F1A37701C4B6D292DB444F1B1507B7EE82D5DAB8107CE1C54DD43E19DF7CC9AD6DDE6AFA0E4E834256955';

OPEN SYMMETRIC KEY ORCHESTRATOR_SYM_KEY DECRYPTION BY ASYMMETRIC KEY ORCHESTRATOR_ASYM_KEY;

with EncryptedPass AS
(
SELECT CONVERT(VARBINARY(MAX), @EncryptedPass , 2) as TheHexPass
)

SELECT convert(nvarchar, decryptbykey(TheHexPass)) AS Password FROM EncryptedPass;

Decrypt All Stored SCO Variables

It might happen that you or someone in your team has stored some variable under Orchestrator Variables as an encrypted variable. And for some reason you want to see the value stored in it now. Once it is encrypted, you cannot remove the check and see the original password. It just shows blank.

In such scenarios the following query might help you find out the decrypted values from the Orchestrator Variables.

use orchestrator;

OPEN SYMMETRIC KEY ORCHESTRATOR_SYM_KEY DECRYPTION BY ASYMMETRIC KEY ORCHESTRATOR_ASYM_KEY;

WITH TheVariables AS
(
SELECT Act.Name, 
        v.Value AS Encrypted, 
        CASE WHEN v.Value like '\`d.%' THEN
        convert(nvarchar, decryptbykey(CONVERT(VARBINARY(MAX), SUBSTRING(v.Value, CHARINDEX('/',v.Value) + 1, LEN(v.Value) - (2 * CHARINDEX('/',v.Value))) , 2))) 
        ELSE
        v.Value
        END AS Decrypted, Act.UniqueID
        FROM [Orchestrator].[dbo].[VARIABLES] v
INNER JOIN [Orchestrator].[dbo].[OBJECTS] AS Act ON Act.UniqueID = v.UniqueID
where Act.Deleted = 0
),
 RunbookPath as
(
select 'Variables\' + cast(name as varchar(max)) as [path], uniqueid from dbo.folders b
where b.ParentID='00000000-0000-0000-0000-000000000005' and disabled = 0 and deleted= 0
union all
select cast(c.[path] + '\' + cast(b.name as varchar(max)) as varchar(max)), b.uniqueid from dbo.FOLDERS b
inner join
RunbookPath c on b.ParentID = c.UniqueID
where b.Disabled = 0 and b.Deleted = 0
)

SELECT F.Path, V.Name as VariableName, V.Encrypted, V.Decrypted  
FROM [Orchestrator].[dbo].[OBJECTS] AS Act
INNER JOIN RunbookPath AS F ON F.UniqueID = Act.ParentID
INNER JOIN TheVariables AS V ON Act.UniqueID = V.UniqueID
where Act.Deleted = 0
ORDER BY F.Path, V.Name

Note : These exercises are from “Linux Training Academy’s” Shell Scripting course.

Exercise 1: Write a shell script that prints “Hello World!” to the screen.

Hint 1: Remember to make the shell script executable with the chmod command.
Hint 2: Remember to start your script with a shebang!

#!/bin/bash
echo "Hello World!"

Exercise 2: Modify the shell script from exercise 1 to include a variable. The variable will hold the contents of the message “Hello World!”.

#!/bin/bash
THE_MESSAGE="Hello World!"
echo "$THE_MESSAGE"

Exercise 3: Store the output of the command “hostname” in a variable. Display “This script is running on .” where “ “ is the output of the “hostname” command.

Hint: It’s a best practice to use the ${VARIABLE} syntax if there is text or characters that directly precede or follow the variable.

#!/bin/bash
THE_HOSTNAME=$(hostname)
echo "The script is running on $THE_HOSTNAME"

Exercise 4: Write a shell script to check to see if the file “/etc/shadow” exists. If it does exist, display “Shadow passwords are enabled.” Next, check to see if ou can write to the file. If you can, display “You have permissions to edit /etc/shadow.” If you cannot, display “You do NOT have permissions to edit /etc/shadow.”

#!/bin/bash

THE_PATH="/etc/shadow"

if [ -e $THE_PATH ]
then
     echo "Shadow Passwords are enabled"

     if [ -w $THE_PATH ]
     then
          echo "You have permission to edit $THE_PATH"
     else
          echo "Tou do NOT have permission to edit $THE_PATH"
     fi
else
     echo "Shadow Passowrds not enabled"
fi

Exercise 5: Write a shell script that displays “man”, “bear”, “pig”, “dog”, “cat”, and sheep to the screen with each appearing on a separate line. Try to do this in as few lines as possible. Hint: Loops can be used to perform repetitive tasks.

#!/bin/bash

ANIMALS="man bear pig dog cat sheep"

for ANIMAL in $ANIMALS
do
        echo "$ANIMAL"
done

Exercise 6: Write a shell script that prompts the user for a name of a file or directory and reports if it is a regular file, a directory, or other type of file. Also perform an ls command against the file or directory with the long listing option.

#!/bin/bash

read -p "Enter a path for file/folder : " THE_PATH

if [ -d $THE_PATH ]
then
        echo "$THE_PATH is a directory"
elif [ -f $THE_PATH ]
then
        echo "$THE_PATH is a simple file"
elif [ -e $THE_PATH ]
then
        echo "$THE_PATH is not a simple file"
else
        echo "$THE_PATH does NOT exist!!"
fi

echo "$(ls -l $THE_PATH)"

Exercise 7: Modify the previous script so that it accepts the file or directory name as an argument instead of prompting the user to enter it.

#!/bin/bash

THE_PATH=$1

if [ -d $THE_PATH ]
then
        echo "$THE_PATH is a directory"
elif [ -f $THE_PATH ]
then
        echo "$THE_PATH is a simple file"
elif [ -e $THE_PATH ]
then
        echo "$THE_PATH is not a simple file"
else
        echo "$THE_PATH does NOT exist!!"
fi

echo "$(ls -l $THE_PATH)"

Exercise 8: Modify the previous script to accept an unlimited number of files and directories as arguments. Hint: You’ll want to use a special variable.

#!/bin/bash

for THE_PATH in $@
do
if [ -d $THE_PATH ]
then
        echo "$THE_PATH is a directory"
elif [ -f $THE_PATH ]
then
        echo "$THE_PATH is a simple file"
elif [ -e $THE_PATH ]
then
        echo "$THE_PATH is not a simple file"
else
        echo "$THE_PATH does NOT exist!!"
fi

echo "$(ls -l $THE_PATH)"

done

The Hive Architecture comprises 3 main components:

1. Hive Client
   This is where the applications get an interface to interact with hive.

2. Hive Services
   Hive services enable the hive interactions by passing them through the hive driver which in turn uses MapReduce.

3. Compute and Storage
   This is the workhorse of the Hive ecosystem which includes the Metastore DB and HDFS storage.

About the Components

Hive Client

Hive client is the interface for different applications and clients which makes it possible for different applications to communicate with Hive.

Hive client comprises Thrift Client, JDBC Client and ODBC Client.

Thrift Client enables Thrift based applications to communicate with Hive. JDBC and ODBC enable several applications and languages to make connections and process with hive.

These clients in turn connect to the Hive Server to make this possible.

Hive Services

Client interactions are made possible by Hive services.

CLI service enables us to interact with Hive via command line.

Web based service makes it possible to have a web based interaction with Hive.

The Hive server service is one of the most important and most used service which most of the Client services connect to.

Hive Storage and Compute

Hive storage used MapReduce in the background which makes it possible for hive to process Big Data.

Hive stores meta information in the Hive Metastore which is the backbone of Hive.

By default, Hive uses Derby DB for storing metadata, but any other RDBMS can be used for this purpose.

However, the data resides on either Local storage or HDFS.

R is in itself a command-line-based scripting language and is a dynamic language (compiled at runtime). Though you can use any text editor to write R scripts, it is way more convenient and useful to use R Studio. In fact, you will almost always use R with R Studio.

R is an open-source. popular and user-friendly (even if you have never done any computer programming) language is used by organizations like Google, Facebook, US DoD, Twitter, TechCrunch, Microsoft, etc.

But, why use R?

Programming and Statistical Features

R is an open-source software created over 20 years ago by Ross Ihaka and Robert Gentleman at the University of Auckland, New Zealand. However, its history is even longer as has derived its features from the S programming language created by John Chambers out of Bell Labs back in the 1970s. S was widely accepted in the world of statisticians. R is actually a combination of S with lexical scoping semantics inspired by Scheme.

R and its libraries implement a wide variety of statistical and graphical techniques, including linear and nonlinear modeling, classical statistical tests, time-series analysis, classification, clustering, and others.

Interfaces

R Studio is the most used interface for R scripting which gives you great flexibility and visual representation of the data you are working with. There are other interfaces available too, including R Tools for Visual Studio by Microsoft and also Rattle GUI, and R Commander. You can even access R functionality from within other programming languages like Python, Ruby, Java, etc.

Flexibility and Community

The functionalities and features of R can be extended with the usage of packages that are very easily available online, developed, and maintained by the community. These packages allow specialized statistical techniques, data handling, data cleaning, graphical devices, import/export capabilities, reporting tools, etc.

These packages are developed by people from different walks of life and knowledge, which include researchers, analysts, scientists, statisticians, and more. The R community is fantastically diverse and engaged. On a daily basis, the R community generates opportunities and resources for learning about R. These cover the full spectrum of training — books, online courses, R user groups, workshops, conferences, etc. And with over 2 million users and developers, finding help and technical expertise is only a simple click away. Support is available through R mailing lists, Q&A websites, social media networks, and numerous blogs.

But the question is, do we really need so many commands when we are talking to Tux the Penguin? The answer is of course No.

So which are the commands that I really need to know about to become a Linux Ninja? Today I came across this awesome “Linux Cheat Sheet” with all the important command-line Linux commands (courtesy: Linux Training Academy).

Download

var myDecimalNum = 10; 
myDecimalNum.toString(16); 

// result : a

var myHexNum = "a"; 
parseInt(myHexNum,16); 

// result : 10

And not just base 16 (hex), you can do it for any base. Try different bases and have fun.

builtins() # List all built-in functions
options()  # Set options to control how R computes & displays results

?NA        # Help page on handling of missing data values
abs(x)     # The absolute value of "x"
append()   # Add elements to a vector
c(x)       # A generic function which combines its arguments
cat(x)     # Prints the arguments
cbind()    # Combine vectors by row/column (cf. "paste" in Unix)
diff(x)    # Returns suitably lagged and iterated differences
gl()       # Generate factors with the pattern of their levels
grep()     # Pattern matching
identical()  # Test if 2 objects are *exactly* equal
jitter()     # Add a small amount of noise to a numeric vector
julian()     # Return Julian date
length(x)    # Return no. of elements in vector x
ls()         # List objects in current environment
mat.or.vec() # Create a matrix or vector
paste(x)     # Concatenate vectors after converting to character
range(x)     # Returns the minimum and maximum of x
rep(1,5)     # Repeat the number 1 five times
rev(x)       # List the elements of "x" in reverse order
seq(1,10,0.4)  # Generate a sequence (1 -> 10, spaced by 0.4)
sequence()     # Create a vector of sequences
sign(x)        # Returns the signs of the elements of x
sort(x)        # Sort the vector x
order(x)       # list sorted element numbers of x
tolower(),toupper()  # Convert string to lower/upper case letters
unique(x)      # Remove duplicate entries from vector
system("cmd")  # Execute "cmd" in operating system (outside of R)
vector()       # Produces a vector of given length and mode

formatC(x)     # Format x using 'C' style formatting specifications
floor(x), ceiling(x), round(x), signif(x), trunc(x)   # rounding functions

Sys.getenv(x)  # Get the value of the environment variable "x"
Sys.putenv(x)  # Set the value of the environment variable "x"
Sys.time()     # Return system time
Sys.Date()     # Return system date
getwd()        # Return working directory
setwd()        # Set working directory
?files         # Help on low-level interface to file system
list.files()   # List files in a give directory
file.info()    # Get information about files

# Built-in constants:
pi,letters,LETTERS   # Pi, lower & uppercase letters, e.g. letters[7] = "g"
month.abb,month.name # Abbreviated & full names for months

Maths

log(x),logb(),log10(),log2(),exp(),expm1(),log1p(),sqrt() # Fairly obvious 
cos(),sin(),tan(),acos(),asin(),atan(),atan2() # Usual stuff 
cosh(),sinh(),tanh(),acosh(),asinh(),atanh() # Hyperbolic functions 
union(),intersect(),setdiff(),setequal() # Set operations 
+,-,*,/,^,%%,%/% # Arithmetic operators 
<,>,=,==,!= # Comparison operators 
eigen() # Computes eigenvalues and eigenvectors 
deriv() # Symbolic and algorithmic derivatives of simple expressions 
integrate() # Adaptive quadrature over a finite or infinite interval. 
sqrt(),sum() 

?Control # Help on control flow statements (e.g. if, for, while) 
?Extract # Help on operators acting to extract or replace subsets of vectors 
?Logic # Help on logical operators 
?Mod # Help on functions which support complex arithmetic in R 
?Paren # Help on parentheses 
?regex # Help on regular expressions used in R 
?Syntax # Help on R syntax and giving the precedence of operators 
?Special # Help on special functions related to beta and gamma functions

Graphical

log(x),logb(),log10(),log2(),exp(),expm1(),log1p(),sqrt() # Fairly obvious 
cos(),sin(),tan(),acos(),asin(),atan(),atan2() # Usual stuff 
cosh(),sinh(),tanh(),acosh(),asinh(),atanh() # Hyperbolic functions 
union(),intersect(),setdiff(),setequal() # Set operations 
+,-,*,/,^,%%,%/% # Arithmetic operators 
<,>,=,==,!= # Comparison operators 
eigen() # Computes eigenvalues and eigenvectors 
deriv() # Symbolic and algorithmic derivatives of simple expressions 
integrate() # Adaptive quadrature over a finite or infinite interval. 
sqrt(),sum() 

?Control # Help on control flow statements (e.g. if, for, while) 
?Extract # Help on operators acting to extract or replace subsets of vectors 
?Logic # Help on logical operators 
?Mod # Help on functions which support complex arithmetic in R 
?Paren # Help on parentheses 
?regex # Help on regular expressions used in R 
?Syntax # Help on R syntax and giving the precedence of operators 
?Special # Help on special functions related to beta and gamma functions

Fitting / regression / optimisation

lm # Fit liner model 
glm # Fit generalised linear model 
nls # non-linear (weighted) least-squares fitting 
lqs # "library(MASS)" resistant regression 
optim # general-purpose optimisation 
optimize # 1-dimensional optimisation 
constrOptim # Constrained optimisation 
nlm # Non-linear minimisation 
nlminb # More robust (non-)constrained non-linear minimisationR

Statistical

help(package=stats) # List all stats functions 
?Chisquare # Help on chi-squared distribution functions 
?Poisson # Help on Poisson distribution functions 
help(package=survival) # Survival analysis 
cor.test() # Perform correlation test 
cumsum(); cumprod(); cummin(); cummax() # Cumuluative functions for vectors 
density(x) # Compute kernel density estimates 
ks.test() # Performs one or two sample Kolmogorov-Smirnov tests 
loess(), lowess() # Scatter plot smoothing 
mad() # Calculate median absolute deviation 
mean(x), weighted.mean(x), median(x), min(x), max(x), quantile(x) 
rnorm(), runif() # Generate random data with Gaussian/uniform distribution 
splinefun() # Perform spline interpolation 
smooth.spline() # Fits a cubic smoothing spline 
sd() # Calculate standard deviation 
summary(x) # Returns a summary of x: mean, min, max etc. 
t.test() # Student's t-test 
var() # Calculate variance 
sample() # Random samples & permutations 
ecdf() # Empirical Cumulative Distribution Function 
qqplot() # quantile-quantile plot

Other Useful Links :

HTTP://WWW.SR.BHAM.AC.UK/~AJRS/R/R-FUNCTION_LIST.HTML

http://www.columbia.edu/~cjd11/charles_dimaggio/DIRE/resources/R/rFunctionsList.pdf https://www.statmethods.net/management/functions.html

chmod: To modify access to files and folders by providing read/write/execute permissions.
chown: To change the ownership of a file or folder.

chmod

There are two ways in which the permissions can be changed/set.

The symbolic method is the one where we mention:

Example :
To remove read and write permission from the group and other of file “myfile.txt”

chmod og-rw myfile.txt

To add execute permission to group of the file “myfile2.txt”

chmod g+x myfile2.txt

The numeric method (my personal favorite) adds up the numbers to determine what kind of permission a file should have:

Now here’s the magic:

1 = execute (1)
2 = write (2)
3 = write (2) + execute (1)
4 = read (4)
5 = read (4) + execute (1)
6 = read (4) + write (2)
7 = read (4) + write (2) + execute (1)

And the only extra thing to remember is the sequence of permission, it’s always
User -> Group -> Others

Example:
To set the permission for a file “myfile2.txt” such as:
a. User has full (read, write, and execute) permission …(read+write+execute = 4+2+1 = 7)
b. Group has read and write permission …(read+write = 4+2 = 6)
c. Others have only read permission …(read = 4)

chmod 764 myfile2.txt

Note: 777 is a very dangerous way to give permission, this gives everyone in the world full control over file.

chown

chown is pretty straightforward. Just remember that the syntax used for ownership is owner:group

Here are a few examples:

To make a user named bob the owner of a folder called MyDirectory:

chown bob MyFolder

To make a group name myGroup the owner of MyDirectory folder (notice the colon before group name, remember the owner:group pattern):

chown :myGroup MyDirectory

To make a user named john and a group named group2 the owner of folder MyDirectory:

chown john:group2 MyDirectory

useradd anupam

But!! the user cannot login yet, because the password needs to be set.
The password must follow the suggested password rule. To set password, just type passwd followed by username and enter the new password ( needs to be executed as root or SU privileges). If you are not a SU then you can only change your password with this command.

You can find the password rules, aging rules etc in the file:

/etc/login.defs

To modify existing user use the command usermod. It has a lot of options like modifying primary group, supplementary group, lock and unlock user account, move or update user home directory etc. Example, to lock user account:

usermod -L anupam

And finally to delete a user, use the command userdel

Pretty interesting eh!

The help command man is more popular and gives concise documentation of how to use a command, whereas the info or pinfo is the GNU documentation which is more detailed.

I personally liked pinfo the most, maybe because I come from a Windows background, but I guess I will use man more 😀

To get help with any of these documentation commands, just write the documentation command followed by the command with which you need help. Example:

man date

or

pinfo date

or

info date

localectl

You can remember it as Locale Control

You can also get more locale information by simply executing the command:

locale

To learn all the possible locale languages type this in terminal:

locale -a

To set locale (to change locale language setting), you can do something like this example:

localectl set-locale LANG=fr_FR.utf8

Also to temporarily to execute any command in a different language, prefix the command with the LANG=<appropriate languagr> <command>. For example:

LANG=fr_FR.utf8 date

To learn more about locale always take help of documentations.