Remove duplicate lines while comparing two files

I’ve been quite busy this whole day with a partially complete database dump and wanted to prepare for tomorrow with some ninja bash voodoo shizzle. I’m doing a braindump here because I know I’ll have forgotten this when I wake up tomorrow :-)

The command stated below was the first working example I’ve gotten together, please let me know if you know a neater / better solution!

The situation:

I’ve got two files. The first file contains lines which need to be deleted from the second line (if they exist there) Continue reading “Remove duplicate lines while comparing two files”

Remove host from SSH KnownHosts file without seeing the hostname

This post is mostly a bookmark for myself. I’ve been using search engines way too often to find this command..

The command to remove a host from a knownhosts file without seeing the actual hostname in the knownhosts file is the following:

ssh-keygen -R HOSTNAME

Continue reading “Remove host from SSH KnownHosts file without seeing the hostname”

Comparing sed stream output in linux

Sed is very very powerful, which is a good thing to be aware of.
I was looking to compare the output of a sed command to the original file before I wanted to execute the sed command directly on the file and came across this handy trick.

It works by using temporary named pipes inside the diff command.

Contents of file:

One
Two
Three
Four
Five

If I just want to remove the line which begins with “Four”, I can check my sed command like this:

joris@beanie ~
$ diff <(sed '/Four/d' numbers.txt) numbers.txt
3a4
> Four

Awesome possum, now I know my sed command won’t destroy anything.

Bash: Shortcuts to your favorite directories with CDPATH

This is a fairly handy trick.

I’m always logging in to different servers, which all have a different location for their logs. With this trick you can login and just type “cd logs” from anywhere.

First we will add our directory to the CDPATH variable:

joris@badattitude /data/share/my_domain
$ echo $CDPATH

joris@badattitude /data/share/my_domain
$ export CDPATH=/data/share/my_domain

joris@badattitude /data/share/my_domain
$ echo $CDPATH
/data/share/my_domain

Then we’ll check if it is working:

joris@badattitude ~
$ cd logs
/data/share/my_domain/logs

joris@badattitude /data/share/my_domain/logs
$

Awesome, we’ve arrived in our logs directory straight from our homedir!

SSH Remote Execute command, multiple command and with interaction

This article shows how to execute remote commands via ssh, but you’ll send the commands from your own shell.

ssh my_server 'ls -l /home/my_home_dir'

This will result in this output:

$ ssh ae2 'ls -lha ~'
total 36K
drwxr-xr-x  2 joris joris 4.0K Jan 23 11:42 .
drwxr-xr-x. 5 root  root  4.0K Jan 23 11:41 ..
-rw-r--r--  1 joris joris   54 Jan 23 11:41 .bash_logout
-rw-r--r--  1 joris joris  507 Jan 23 11:41 .bash_profile
-rw-r--r--  1 joris joris  213 Jan 23 11:41 .bashrc
-rw-------  1 joris joris   51 Jan 23 11:42 .history
-rw-r--r--  1 joris joris  171 Jan 23 11:41 .kshrc
-rw-r--r--  1 joris joris  375 Jan 23 11:41 .profile
-rw-r--r--  1 joris joris  153 Jan 23 11:41 .vimrc

What’s even better, is that you can run multiple commands separated with a semi colon, like this:

ssh my_server 'ls -l /home/my_home_dir;whoami'

And the best trick is this one, user input with an interactive command, sending input and output back and forth!

ssh -t my_server 'vi ~/.bash_profile'

Find Java JRE location on Ubuntu Linux

Everybody knows that the java executable is located in /usr/bin/java , but what if you need the JDK / JRE location itself?

Just using “whereis” will not get you there, that will point you to the /usr/bin/java point.

joris@howlingmad: ~_011

 

So, let’s find out a but more about /usr/bin/java:

ls -l /usr/bin |grep java

joris@howlingmad: ~_012

 

Awesome, this will lead us somewhere, it’s a symlink to /etc/alternatives/java

So let’s do the same there:

ls -l /etc/alternatives/ |grep java

And we’ve hit the jackpot, among the lines here, there’s a bunch of lines pointing us to the JRE location:

joris@howlingmad: ~_013

 

As you can see in the screenshit, our java executable within the JRE location is:

/usr/lib/jvm/java-7-oracle/jre/bin/java

Which means the JRE location is:

/usr/lib/jvm/java-7-oracle/jre

 

 

 

CHere Bash Here without Admin Rights – CYGWIN

I wanted to add a “Bash Here” context entry when I right click on a directory:

Screenshot - 28-2-2014 , 13_01_20

This Bash Here would open that directory in Cygwin (mintty).

My current PC is fairly regulated, so without admin rights and thus I can’t use the CHere option which I would normally use.

I just created the registry keys via regedit and that works fine.

You can use two options:

  1. Import the registry file mentioned below
  2. Create the keys yourself

1. Import the registry file

Create a file called cygwin_bash_here.reg with the following contents:

Windows Registry Editor Version 5.00

[HKEY_CURRENT_USER\Software\Classes\Directory]

[HKEY_CURRENT_USER\Software\Classes\Directory\shell]

[HKEY_CURRENT_USER\Software\Classes\Directory\shell\bashhere]
@="Bash Here"

[HKEY_CURRENT_USER\Software\Classes\Directory\shell\bashhere\command]
@="C:\\cygwin\\bin\\mintty.exe"

Save the file and double click on it to import it to the registry.

2. Create the keys yourself

  1. Open the registry editor: Start – Run – “regedit”
  2. Open the following path: HKEY_CURRENT_USER\Software\Classes
  3. Create key: “Directory”
  4. Inside Directory, create key: “shell”
  5. Inside shell, create key: “bashhere”
  6. Inside bashhere double click on the standard key and insert text “Bash here” without quotes (This is the text which is displayed in the context menu.
  7. Inside bashhere, create key: “command”
  8. Inside command, double click on the standard key and insert text “C:\cygwin\bin\mintty.exe” without quotes. Important: This is the command which is run. Please change directory to your cygwin directory

Screenshot - 28-2-2014 , 13_16_02

Conclusion

Both options will create a context menu item which opens Cygwin at your current windows directory!

Cygwin – Your group is currently mkpasswd

This message occurs every time you start your freshly installed Cygwin when you’re logged in as a domain user.

Your group is currently "mkpasswd".  This indicates that your
gid is not in /etc/group and your uid is not in /etc/passwd.

The /etc/passwd (and possibly /etc/group) files should be rebuilt.
See the man pages for mkpasswd and mkgroup then, for example, run

mkpasswd -l [-d] >> /etc/passwd
mkgroup  -l [-d] >> /etc/group

Note that the -d switch is necessary for domain users.

Important: You need to install Cygwin with the user you are logged in with.
Tip
: Remove the word “setup” from the cygwin executable to be able to install it without administrator privileges. (e.g. setup-x64.exe should be renamed to cygwin-x64.exe)

  • mkpasswd -l only shows my local users, and not the domain user I’m logged in with, so that does not solve this.
  • mkpasswd -l -d get an enormous amount of users because it tries to replicate my whole organisation, which is not necessary.

We just need our current user ( mkpasswd -c ) to be sent to the /etc/passwd and /etc/group files, to do this, we use this command:

Solution:

mkpasswd -c >> /etc/passwd
mkgroup -c >> /etc/group

After that, our current account is added to both /etc/passwd and /etc/group and the annoying greeting message is gone!

SSH through a proxy to a remote server

I wanted to SSH into my home server from my workplace but I couldn’t reach it directly because of the way the network was set up.

As it turns out it is quite easy to do by using the corkscrew program.

Edit ~/.ssh/config and add the following lines:

Host home joris.his.homeserver.com
    Hostname joris.his.homeserver.com
    User joris
    ProxyCommand corkscrew proxyserveraddress proxyserverport %h %p

The most important part is the ProxyCommand, this lets your ssh client know that it should use corkscrew as a proxy to your host. %h means the host of your remote server, %p means the port of your remote server.

Authentication – Subversion on command line will not remember credentials

I ran into this issue today on my command line SVN client.

Every time I ran the SVN command against my repository, it asks for my password. It does remember the username but doesn’t store the password.

There are a couple of settings to check in two different files:

  • .subversion/config
  • .subversion/servers

.subversion/config

The config file contains a setting which sets the password store you will be using. We need to disable all password stores and use an empty list, which is done by uncommenting (or adding) the next line

password-stores =

.subversion/servers

The servers file contains settings which allow you to save your passwords in general and to save the passwords in plaintext (please be careful when choosing this option!)

This file is divided in sections which are set with the [] brackets. In the [global] group you should uncomment (or add) the following lines:

store-passwords = yes
store-plaintext-passwords = yes