Grant McWilliams

grep a file in python

Grep in Linux is an amazing tool. Sometimes I just want the simplicity of grep in Python. This isn't the equal to grep but it will take a regular expression as an argument (two examples shown) and give a return code of success if found or None if not. It will also return the line just like grep. 

 

#!/usr/bin/env python
 
import re

def grep(patt,file):
    """ finds patt in file - patt is a compiled regex
        returns all lines that match patt """
    matchlines = []
filetxt = open(file) lines = filetxt.readlines() for line in lines: match = patt.search(line) if match: matchline = match.group() matchlines.append(matchline) results = '\n '.join(matchlines) if results: return results else: return None # Example use textfile = "/etc/hosts" file = open(textfile) criteria = "localhost" expr = re.compile(r'.*%s.*' % criteria) # finds line that starts with anything, ends with anything and has criteria in it #expr = re.compile(r'[0-9].*filename:(%s)\schecksum:.*result: (.*)' % criteria) # more complex example # using return code if grep(expr, file): print criteria + " is in " + textfile else: print criteria + " is not in " + textfile file.seek(0) # rewind file for next test # printing all matching lines results = grep(expr, file) print results file.close()

Since Python is so picky about indention (drives me crazy) you can download grep.py from my downloads section.

Add a comment

Pumpkin Smackdown!

Many years ago when my daughter was first born we'd receive WIC coupons from the local office for food to help with nutrition and such. Those coupons were only good for certain things but you could buy just about anything from the local farmers market with them. Breezing the halls of the farmers market was an interesting thing because at the time I didn't do a lot of cooking and the only thing the farmer's market sold was items that needed to be cooked. As fate would have it I made a choice and bought a pumpkin. Once home I had to figure out what to do with it so I made Pumpkin bread and the rest as they say is history.

I've now been making Pumpkin bread every fall for 20 years without missing a single season. Since then I've also learned a great deal about food and Pumpkins specifically. As most people I started out buying Jack-o-Lantern pumpkins because this is the quintessential pumpkin that everyone recognizes. Little did I know they don't make great food. Cooks Illustrated a magazine I respect greatly maintains the idea that it's just not worth the effort to cook raw pumpkins but I beg to differ. Had I stopped at the Jack-o-Lantern pumpkins I would agree but there are better pumpkins out there when you have food in mind which is why we're here today.

  

Pumpkin Varieties

Shortly I'll be outlining the 3 pumpkin varieties worth considering for food and referencing the pumpkin people usually try to make food out of unsuccessfully – the Jack-o-Lantern. There are many many varieties of pumpkins but half of them are branches from the Jack-o-lantern tree so we'll cover them together. Then there's the smallish Sugar Pie Pumpkin, the Long Island Cheese Wheel and the Cinderella. The latter two have limited availability although popularity of the Cinderella seems to be on the rise if only slightly.

 

Testing Criteria 

Flavor: Of course flavor will be number one. Contrary to popular belief not all pumpkins taste the same and why would they? Not all squash taste the same so it makes sense. 

Texture: Texture is important when making puree out of the meat.

Cookability: This is more important than you think. I've cooked pumpkins every possible way looking for the method that gives me the most meat, the best flavor and texture. Some pumpkins are more cookable than others. Pumpkins that are too small or too large are difficult to cook while either maintaining flavor or getting a decent ratio of meat to work involved. See my method on how to cook a pumpkin a bit later in this article. 

Longevity: Because I only use fresh pumpkin in my bread and I worry about availability more than most. Most pumpkins disappear from the store about Halloween time. This is a problem for me because I like cooking pumpkin bread for more than the two weeks leading up the Halloween. There are vast differences between pumpkins in regard to longevity.

Availability: Availability is important because if you can't buy the variety to begin with it doesn't matter how good it is. Some pumpkins like Jack-o-Lantern are always available around Halloween but ONLY right up to October 31st. Try to get one the next day. Others just aren't distributed or grown much.

Cost: How much do I usually have to pay?

  

Add a comment

Read more: Pumpkin Smackdown!

Create custom templates

Prerequisites

  • XCP/Xenserver

 

Creating a template

The following is how to create new custom templates based on existing templates in Xen Cloud Platform. 

1. Get the template UUID that we want to use as our base. As usual just copy and paste the line in yellow into a root terminal on your XCP host.

xe template-list | grep -B1 name-label | awk -F: '{print $2}'

The output should look like this..

 

 688c625b-93b8-8e66-62e5-4542eca1e597

 Red Hat Enterprise Linux 6 (32-bit)

 

 c4e28252-030f-524a-c5d8-7da85df3ccf5

Windows Server 2003 (64-bit)

......

 

Scroll through the list and find the template you want to clone then copy and past it's UUID number ie.  688c625b-93b8-8e66-62e5-4542eca1e597. Choose a new name for your custom template and enter the following line with the UUID of the template you want to clone and the name you want it to have.

 

xe vm-clone uuid=<UUID> new-name-label="<NAME>"
xe template-param-set uuid=<UUID> other-config:install-methods=http,ftp,nfs other-config:default_template=true

Now you should have a new template of your own that you can customize. More after the jump.

Add a comment

Read more: Create custom templates

How to manage VM images on XCP

Once you've created a VM from one of my other tutorials (CentOS 64bit/CentOS 32 bit) you may want to make copies/clones and/or backup the images. There are several ways that you can make a backup copy of a VM all with subtle differences so we'll cover them below. 

 

Copying a Virtual Machine

The following is how to make a full copy of a VM on Xen Cloud Platform. When you do a full copy it does just that, it creates a new VM and makes a full copy of the disk.

Syntax from the XenServer Administrators Guide

vm-copy new-name-label=<name_for_copy> [new-name-description=<description_for_copy>] 
[sr-uuid=<uuid_of_sr>] [<vm-selector>=<vm_selector_value>...]

This may need a bit of explaining. You need to know a couple of items before you can make a copy of a VM.

  1. The Name or UUID of the old VM
  2. The Name of the new VM
Optionally
  1. The Description of the new VM
  2. The Storage Repository to hold the new VM

 

The simplest method of copying a VM would be to just accept the defaults.

 

[root@cloud1 ~]# xe vm-list
uuid ( RO)           : adde907a-3b85-80e5-e7c9-47718dd1d55b
     name-label ( RW): baseimage
    power-state ( RO): halted
[root@cloud1 ~]# xe vm-copy name-label=baseimage new-name-label=baseimage-copy
 

This process may take a few minutes depending on the size of the disk image. It's worth noting that it's more reliable to use a VM's UUID to identify it than it's name and the reason for this is that XCP allows you to have more than one VM in a pool with the same name. The line above would look a bit different if we'd used the VM's UUID.

[root@cloud1 ~]# xe vm-list
uuid ( RO)           : adde907a-3b85-80e5 -e7c9-47718dd1d55b
     name-label ( RW): baseimage
    power-state ( RO): halted
 [root@cloud1 ~]# xe vm-copy vm-uuid=adde907a-3b85-80e5-e7c9-47718dd1d55bnew-name-description="Copy of baseimage" new-name-label=baseimage-copy new-name-label=baseimage-copy

 

Using the UUID isn't quite as people friendly as name-label because UUID's are a bit scary looking. I usually use name-label to identify VM's unless something funky is going on then I'll switch over to UUIDs. For instance I rebooted a VM by name-label and was logged into the whole time and it clearly didn't reboot. While troubleshooting I realized I had two VMs with the same name-label and XCP rebooted the one I wasn't logged into. Below is an example of multiple VMs using the same name-label.

  

[root@cloud1 ~]# xe vm-list name-label=baseimage-copy

uuid ( RO)           : 7062651a-6c05-ac3b-fe71-5dd9b6ca2c18

     name-label ( RW): baseimage-copyThe simplest method of copying a VM would be to just accept the defaults.

 

 

 

[root@cloud1 ~]# xe vm-list

uuid ( RO)           : adde907a-3b85-80e5-e7c9-47718dd1d55b

     name-label ( RW): baseimage

    power-state ( RO): halted

[root@cloud1 ~]# xe vm-copy name-label=baseimage new-name-label=baseimage-copy

 

This process may take a few minutes depending on the size of the disk image. It's worth noting that it's more reliable to use a VM's UUID to identify it than it's name and the reason for this is that XCP allows you to have more than one VM in a pool with the same name. The line above would look a bit different if we'd used the VM's UUID.

 

    power-state ( RO): halted

 

uuid ( RO)           : fce951a3-1105-6e36-b6db-09ba3fd37180

     name-label ( RW): baseimage-copy

    power-state ( RO): halted 

 

The other instance where I use UUIDs would be scripting. If I'm managing VMs using a shell script I'll always use UUIDs since shell scripts don't care about pretty name labels and UUIDs are more reliable.

You may want to add a description to the VM as well for future reference.

[root@cloud1 ~]# xe vm-list
uuid ( RO)           : adde907a-3b85-80e5-e7c9-47718dd1d55b
     name-label ( RW): baseimage
    power-state ( RO): haltedthe Man, the Myth, the Legend
 [root@cloud1 ~]# xe vm-copy name-label=baseimage new-name-label=baseimage-copy new-name-description="Copy of baseimage"
[root@cloud1 ~]# xe vm-list name-label=baseimage-copy params=name-description,name-label 
name-label ( RW)          : baseimage-copy
    name-description ( RW): Copy of baseimage
  
 
 
Sometimes you need to specify the Storage Repository to copy the image to. In the examples above the disk image will be copied to the DEFAULT Storage Repository so normally you don't have to specify. However there are several cases where you may want to specify. An example of this would be if the storage repository you want the VM copied to isn't default.
[root@cloud1 ~]# xe sr-list 
uuid ( RO)                : 92bf5d33-a164-d75c-26c0-3f53f0490ba0
          name-label ( RW): iSCSI_Disk2
    name-description ( RW): SSD on cloud0
                host ( RO): <shared> 
                type ( RO): lvmoiscsi
        content-type ( RO): user
[root@cloud1 ~]# xe sr-list
uuid ( RO)                : 92bf5d33-a164-d75c-26c0-3f53f0490ba0
          name-label ( RW): iSCSI_Disk2
    name-description ( RW): SSD on cloud0
                host ( RO): <shared>
                type ( RO): lvmoiscsi 
        content-type ( RO): user
[root@cloud1 ~]# xe vm-copy name-label=baseimage new-name-label=baseimage-copy new-name-description="Copy of baseimage" sr-uuid=92bf5d33-a164-d75c-26c0-3f53f0490ba0
 
 This will copy the VM to the Storage Repository with the UUID of 92bf5d33-a164-d75c-26c0-3f53f0490ba0. Be careful with this though. If the Storage Repository isn't accessible by that host ie. it isn't shared then the VM won't be able to start. 
 

Cloning a Virtual Machine

Cloning a VM is very similar to Copying it with the exception of it being much faster and you can't choose your Storage Repository. You can't specify the Storage Repository because it creates a Copy on Write instance of the original disk. This means the clone goes very fast due to it not writing very much data. As a VM changes the contents of the disk by adding/removing files the changes are stored in the Copy on Write clone and not the original disk. This means you can create them very fast but you can't clone to a new Storage Repository.

[root@cloud1 ~]# xe vm-list
uuid ( RO)           : adde907a-3b85-80e5-e7c9-47718dd1d55b
     name-label ( RW): baseimage
    power-state ( RO): haltedthe Man, the Myth, the Legend
 [root@cloud1 ~]# xe vm-clone name-label=baseimage new-name-label=baseimage-copy new-name-description="Copy of baseimage"
[root@cloud1 ~]# xe vm-list name-label=baseimage-copy params=name-description,name-label 
name-label ( RW)          : baseimage-copy
    name-description ( RW): Copy of baseimage

 

Something to keep in mind. If you clone VM1 to VM2, then clone that to VM3 and so on you will eventually have a great deal of overhead writing to the disks. Each time a VM wants to write to disk Xen Cloud Platform has to check each disk image to see what's changed. Therefor it's recommended to clone the original and not clone a clone. If you have multiple levels of clones you can do a vm-copy to make a fresh new VM disk to restore performance.

 

Add a comment

Read more: How to manage VM images on XCP

50% off of free!

I often take photos of humorous department/grocery store signs. This one I took years ago but missed putting it up. This is in the famed Galleries Lafayette in Paris, a store that charges enough for it's clothes to pay for a personal mathematician to go around and correct signs. It appears he was off that day.

So this 50% off of free, is that before or after taxes? And also is there a chance the price will go down further because I've heard rumors that next week it will be 60% off of free.

 

I'm sure this is a translation error but still it's funny. Free minus 50%, discount already deducted. You know because it's so hard to calculate. 

While we were there we saw a dress that was 9,000 Euros, about $13,500 U.S. dollars at the time. Too bad that wasn't 50% off of free. 

Add a comment

New site status update (updated)

As I work through functional bugs I'm able to file them away on the site migration. Up until yesterday RSS feeds had stopped working. This had to do with some very aggressive caching I was doing. This has been fixed by turning the caching off.

 

What's still broken/sucks.

  1. Photo gallery theme still so ugly it's mother doesn't even appreciate it
  2. Disqus comments broken. Not sure if it's a bug in SP comments (it's sending a relative URL to Disqus) or a site misconfiguration
  3. Old Disqus comments are not mapped to the new URLs. I have the URL map in csv and now just need to write a script that dumps the new URLs from mysql, greps for the article name and rewrites the URL. This will then be uploaded to Disqus. Sort of moot since they don't work anyway.
  4. I need an old URL to new URL map for mod_rewrite so all links out there pointing to the old URLs still work
  5. My visitor counter either doesn't seem to be reporting the same statistics as the old one or Google is severely punishing me (perhaps the latter) for changing all the URLs it had cached. If it's the latter it will go away at some point.
  6. My visitor counter and who's online module don't agree. There will be 50 people online but the visitor counter says there's only been 8 people all day. How is that possible?
  7. Lots of CSS work still to be done. I'd like to have an automatic white space border around photos plus a nice charcoal gray line. Also the right hand column of links will probably be moved back to the left but I need to write the CSS for that. I'd also like to move the search bar down to the menu bar, re-enable breadcrumbs and work on the surrounding whitespace which I don't like. I'm also going to change the main body text from 12 points to 13 as soon as I figure out who's overriding whom in the CSS.
  8. Recipebook is completely offline. I'm building a new one with SobiPro. This also means I'll need to purchase some modules to get the same functionality that I had before. 
  9. Restaurant Reviews are also offline for the reason stated in number 7.
  10. No file download manager to manage all downloadable scripts, templates etc..
We'll get there. Once all functionality has been restored I'll be working on performance. I'd really like to get all my caching back on. I'm also thinking of running a Varnish server in front of the website.

New site rollout!

 After a month (count them, 30 days!) of working on the new site I've rolled it out. The amount of work involved in completely recreating a working production website that's been online for years is quite a lot. Below I've created a table to give you an idea of what was done. 

 

  Old Site  New Site 
Host Operating System     CentOS 5.3      Xen Cloud Platform 1.1    
Hypervisor  Xen 3.4.0  Xen 3.4.2
Virtualization Stack  OSS Xen  XAPI  
Guest Operating System  CentOS 5.6  CentOS 6.1
Memory   2 GB  7 GB
Guest Operating System        CentOS 5.6  CentOS 6.1
Storage Subsystem   File based local disk       iSCSI SAN
Content Management System    Joomla! 1.5 Legacy mode   Joomla! 1.7 Native mode
Blogging component  Superblogger  K2
Directory component  Sobi2  Sobi Pro
Photo Gallery  Gallery 2.1  Gallery 2.3
Gallery Bridge  g2bridge  Jfusion
Gallery Theme  PGtheme  ?
Joomla Theme  Modified JA-Purity  Modified JA-Purity ii

 

If you look at this chart it may not seem like it was that big of a deal however, it was.

Hardware:

I built new hardware based around AMD's Phenom II hexacore CPU. The old server has been awesome but just buying ram to upgrade it was getting difficult and expensive. Also since that server is a real server with real jet airplane speed fans it has to be in remote parts of the building otherwise you'd need new eardrums. Because of this the network connection was tunneled across a wifi link. Not ideal but it's always worked. In order to gain reliability the new server is wired directly into the ISPs connection. I'm looking into a dedicated connection and another speed increase.

Virtualization Stack:

Open source Xen has served me well for many years but as I move everything over to clouds I've been rolling out more Xen Cloud Platform installations. I could have cheated and just migrated the old disk image but I really wanted to take the time to create a new fresh disk image and also roll out CentOS 6.x. .While going through this process I created new XCP howtos on how to install CentOS 6.x on Xen Cloud Platform.

Content Management System:

Once I had that done and the bugs worked out I created I migrated Joomla! 1.5 to Joomla 1.7. This is NOT an upgrade, it's an entire migration. There are some scripts that help you bit in reality what they do is move your articles over and not much more. Because ALL of my components, modules and plugins stopped working on Joomla 1.7 I had to find replacements. The big one was Superblogger which doesn't have a Joomla 1.7 version.

After a great deal of research I went with K2 made by the same folks as Superblogger. K2 isn't an exact replacement since Superblogger is made for blogging and includes trackbacks, Disqus comments and some theming. K2 is really a component that replaces a lot of Joomla! Ironically a lot of it doesn't need to be replaced and I hope in the future we'll see a newer leaner K2. For instance K2 provides nested categories and access control lists, two features not included in Joomla 1.5. However, Joomla! 1.7 has both so K2 doesn't really need them anymore. Using K2 is really an all or nothing proposal since it takes over all of Joomla! categories/articles, access control and some theming. It does NOT include Disqus commenting but there is an extra plugin for that... which unfortunately doesn't work on Joomla! 1.7 (common theme here) so I went with SP Comments.

I used Sobi2 for my restaurant reviews and recipes which as you may have guess also didn't work. The folks that make Sobi have come out with Sobi Pro which won't import Sobi2 data. Sobi Pro looks pretty good and will allow me to get around the hack I had to use to run two copies of Sobi2 in one instance of Joomla!. Most of Sobi Pro's additional functionality though is commercial so it looks like I'll be paying out some money. One example is a tie in with Google maps. It would be nice to do a restaurant review with address and have a Google map appear automatically. 

My old photo gallery was Gallery2 embedded in Joomla! 1.5 using the g2bridge component which as you may have guessed isn't available for Joomla 1.7. I found a replacement in Jfusion which not only will bridge Gallery2 but PHPbb and a bunch of other software. However, I can't figure out how to provide a menu item that links to a specific gallery. I could do this with g2bridge. I also used g2image so I could insert Gallery2 images in articles. I don't have an equivalent with JFusion. I used the PGtheme theme for my embedded Gallery2 which (wait for it....) doesn't work with the downloadable version of Gallery2!  Gallery3 is out but JFusion doesn't support it so the latest version of Gallery2 is what I will use. I haven't even begun writing a new theme for it so it's currently pretty ugly. 

My old Joomla 1.5 theme was JA-Purity which I've always liked. I hacked it up pretty good and it eventually started having problems. JA-Purity doesn't work on Joomla 1.7 (why would it???) but the folks that developed it came up with JA-Purity ii for Joomla! 1.7 which I've also spent quite a lot of time hacking on to get it near where I want it.

In addition, my contact module had to be replaced, my visitor counter had to be replaced, all articles tagged for the new Tag Cloud (which is nice) and more. I've spent so much time on this migration that I don't even remember what I've done anymore. I wished now that I'd documented it all. 

What's left?

  1. Keep working on the Joomla 1.7 theme. There's a ton of cleaning up and redesigning I want to do.
  2. Get serious about SEO. I have SEF working as well as I can so my URLs are better. 
  3. Crate a Gallery2 theme
  4. Create the entire directory structure in Sobi Pro so I can get back to writing reviews and putting recipes up
  5. Find a way to link to specific Gallery2 gallery.
  6. Create metadata for everything
  7. Finish tagging the last 100 articles for the tag cloud
  8. Find a downloads component that works with Joomla 1.7
  9. Find a way to handle weblinks in K2 instead of Joomla! articles
  10. Get an autotweet/facebook plugin
  11. Install tracking/analytics component of some sort
  12. Experiment with putting Varnish between the webserver and the ISP (which is why there's no login form on the front page)
  13. Create a URL map of my old comments mapped to the new URLs for Disqus. Until then old comments won't link properly
  14. Experiment with a server side PHP accelerator
  15. Perhaps look into Adwords/Referrals
  16. Create content!

 

That's it! It's been a lot of work to get a new site that does exactly what the old one accomplished and it's still not quite there.

Recession Chef: Eggs, Bacon and French Toast with pizzazz

Sometimes I want to be fancy. Food is fun. I could just throw some beaten eggs in a pan until they're solid, toss some egg soaked bread on the grill and fry up some bacon but what fun is that? 

My daughter had been asking me for French Toast so I gave in today and mixed an egg, some flour, sugar, salt and milk together to slather onto Texas Toast. The eggs were totally French style done in a double boiler. I didn't have any morels or truffles so I just added a touch of vanilla to them. The Bacon was coated with cracked pepper. 

The fancy part comes with the preparation. I cut off the top of the egg with your typical $20 "cut a muffler pipe" knifeset by ginsu (I think they're called Master Chef now) serrated knife. I washed out the eggshell nicely and filled it with the French style egg custard. 

A strip of bacon pressed flat while fried worked nicely as the dipper utensil for the eggs. Overall this fancy breakfast didn't cost any more than the traditional "slap it on a plate" method but was more fun for sure. 

You don't have to go out to breakfast and spend a fortune for something with pizzazz. 

Add a comment

Latest Comments

{loadposition content_comments}

Add a comment

Fix the XCP expired license issue

If you're running virtually any version of Xen Cloud Platform you may have run into this error message.

Your license has expired.  Please contact your support representative.

It's not really possible to have an expired license on Xen Cloud Platform (XCP) since it's FREE. It's just a regressive bug that has been very stubborn. However, until they fix it for real in XCP 1.5 you'll need follow the steps below.  

Open a root terminal on the XCP host and copy and paste the commands below.

 

service xapi stop;sleep 5
NEXTMONTH=`date --date="next Month" '+%Y%m%d'`
sed -i "s/\(expiry.\{3\}\)[0-9]\{8\}/\1$NEXTMONTH/g" /var/xapi/state.db
service xapi start
echo done

 

The last line is only to get all the important lines to run automatically. If you don't hit enter it doesn't hurt anything.  You could also copy and paste these lines into a script and have it run as a cronjob. Because XCP doesn't like you bumping it's "evaluation license" out more than 30 days you might want to run the cronjob once a week to make sure your license doesn't lapse while you're waiting for the cronjob to run

Add a comment