Below you will find a short refresher on Linux commands that you will use or might find useful while running through the labs. Below that is a short command line focused guide on debugging issues in OpenStack.
Basic Commands
Directory listing:
ls
short listls –l
long list
Change directory:
cd
change directorypushd/popd
push and pop directory stack
Command history:
history
list history!!
previous command!$
last element of previous command!#
run command from history numbered # (e.g. !42)
Become root:
sudo –H bash
sudo su –
Editing Files
Edit file vi {file_name}
open file in editor
VI commands:
:
enter command mode:wq
write and quit:q!
quit without writing changes:w
write and keep editing/{string}
search forward?{string}
search backward:1,$ s/{regex}/{substitution}/g
globally (across the file and across a line) search and replacei
insertc$
change to end of linecw
change ‘word’
Debugging Support
Use of these set of commands below is explained in greater detail on the Debugging OpenStack section on this page.
screen
screen –L
start screen, log output<ctrl>-a c
create another screen<ctrl>-a <space>
swap screen sessions (in order)
Log files
- Open multiple screen sessions
- In one, “tail” the log file:
tail –f /var/log/glance-api.log
tail and follow the file<ctrl>-c
exits out of the tail command
- Logging may be low level, look at .conf file
/etc/{project}/*conf
, turn on Verbose and Debug options. - Restart services if changing logging levels
Quick Tip
Making your life a bit easier when working with vi (in the VNC) From the command line in your terminal:
echo ‘set background=light’ >> ~/.vimrc
screen: screen is a useful tool as it allows you to have multiple “views” in a single terminal window, and lets you switch between them with a keystroke rather than opening multiple windows and point/clicking between them. In addition, screen can log the output of a terminal session to disk, making it useful for capturing the output of failures for later review.
Start screen and log the output:
screen –L
Once started, you can create a new screen with:
<ctrl>-a c
Which is hold the control key, and press a, release the control and a, and the press c.
Move between the screens with
<ctrl>-a <space>
Screen does live within a single “view” of the terminal, so to look at the scrollback buffer, you have to :
<ctrl>-a <esc>
And then use the arrow keys on your keyboard to scroll up (-b scrolls back a page, -f scrolls forward)
Get help for screen (many more capabilities):
<ctrl>-a ?
log files: There are numerous log files per service, and often the error will be exposed in the log file output. Each project will have a log directory in /var/log{service}. Since services have different components that operate in similar but not always equivalent fashion, it’s best to use screen to open a session per log file so you can watch the entire set of possible outputs to find the system that isn’t responding appropriately. You can follow the logs with the “tail” command such as:
tail -f /var/log/{SERVICE}/api.log
When you are done following the file, you can exit out with:
<ctrl>-c
- Logging may be low level, look at .conf file (/etc/{service}/*conf), turn on Verbose and
Debug options. (e.g. etc/glance/glance.conf)
- Restart services if changing logging levels
- Now that we know how to watch the logs and can see what’s going on, we can try to spot the errors both in the .conf file (just by reviewing the actual lines typed in. Do you understand them?):
Usual issues are that AMQP isn’t connected and eventually quits trying to connect (usually in the API log). The same is true for mysql mis-configurations. If an error is found it’s best to restart all of the services in that project group (e.g. all the nova processes) in case there’s a non-obvious interaction with Rabbit or Mysql buried in the application components themselves.
You can check through the following configurations in /etc/{service}/{service}.conf
the keystone API URI/URL components and port.
The hostname provided for all the different services
Check target URI/URL components in .conf file
Validate endpoint in keystone. You can do this via the {service} cli tool, for example:
glance --debug image-list
Should resemble out something like:
glance --debug image-list
curl -i -X GET -H 'Accept-Encoding: gzip, deflate' -H 'Accept: */*' -H 'User-Agent: python-glanceclient' -H 'Connection: keep-alive' -H 'X-Auth-Token: {SHA1}73459d07b6cf38f67503dc2ec9090c48c479c8b7' -H 'Content-Type: application/octet-stream' http://c240-1.onecloud:9292/v1/images/detail?sort_key=name&sort_dir=asc&limit=20
REST Commands :
The cli with –debug also resemble out the correctly focused ReST calls (with curl as the application already even) that you can generally pasted into your console to see what’s happening.
Rapidly restarting services
If you need to restart all of the nova or neutron services, you can do so quickly on Centos as follows:
for name in (cd /etc/init; ls *{service}’ | cut -d. -f1)
; do service ${name} restart; done