Static site, Python search

So now that I have my Python program searching the way I want. I need to be able to use it on the web. Which reminded me of a silly little Python 2 program I wrote a long time ago that I had on my site at one time. The program is called and can be accessed here. It maybe silly but also interesting because it tells you how to pronounce very large numbers. For my search program, this program is a good reference, because it’s simple, it doesn’t use any complicated web framework. It only uses cgi (hopefully available in Python 3) to get the input from the form. I just manually write out HTML. No PHP needed!

Static Website search

Looking for a way to add “search” to a static site. I considered using grep offline on the html pages. Then referencing the web page. But grep gave me problems. I have to assume that with something used as much as grep, the problems were due to my misuse. Be that as it may, if I can’t get it to do what I want then for this purpose it is useless


logfile/posts$ grep -il "vm/" *.html

logfile/posts$ grep -i "vm/" *.html
339.html:<h1>VM/370 emulation</h1>
339.html:   Ran VM/370 in a IBM S/370, ESA/390, and z/390 hardware emulator on Linux.
345.html:   Today I'm more comfortable using MVS in Hercules emulation than VSE and VM. Mostly because MVS didn't take as much work as VM/DOS to actually do something useful. It was more useful as they say...out of the box.
457.html:<h1>IBMs VM/370</h1>
457.html:   Continuing retro. Now using the hercules emulator for IBM Mainframes and also the 3270 terminal emulator. Here's VM/370,  IBM's first version, released in 1972...way before Linux knew what a VM was or, for that matter, the world knew what Linux was. This is very much like the VM systems I was employed to maintain...

Right now I have all posts in one directory, but I’m considering breaking them up in subdirectories by year. So I need to be able to search the directories recursively.

So much for recursive search working
This from the help….
-r, --recursive           like --directories=recurse

logfile$ ls -la
total 116
drwxrwxr-x  3 bill bill  4096 Aug 28 14:45 .
drwxr-xr-x 35 bill bill  4096 Aug 25 18:52 ..
-rw-rw-r--  1 bill bill  2846 Aug 22 18:23 about.html
-rw-r--r--  1 bill bill  3832 Aug 28 14:45 grep.txt
-rw-r--r--  1 bill bill 59270 Aug 25 21:59 index.html
drwxrwxr-x  2 bill bill 36864 Aug 25 21:59 posts	Subdirectory one up

logfile$ grep -r "vm/" *.html
logfile$ 		Nothing returned!

I read something saying grep has a problem with wildcards, so...

logfile$ grep -r "vm/" .
logfile$		Again nothing returned!

I read that grep has something to do with REs  global regular expression
So why did “VM/” work in the 1st examples?

However (finally on my own research) I suspected the slash…
As you can see below “escaping” the forward slash worked, However I didn’t escape it above...and it worked! That’s inconsistent!

logfile$ grep -ril "vm\/" .

Finally. I read you can use the “-F” switch to do a regular, non-regex search. But that didn’t work either!

Supposedly you can avoid regexs by using -F. However…

logfile$ grep -r "vm/" .
logfile$ grep -Fr "vm/" .
logfile$ grep -r --fixed-strings "vm/" .
logfile$ grep -F -r  "vm/" .

Finally I just wrote a ~40 line Python program that does what I want. As you can see it recursively searched 2 directories. Now was that so hard?

(base) bill@bill-MS-7B79:~/Mystuff/Python3$ ./ 
Enter string to search for: vm/
Searching for: vm/
Found in: logfile/posts/342.html
Found in: logfile/posts/453.html
Found in: logfile/posts/336.html
Found in: logfile/index.html

Files searched  458
(base) bill@bill-MS-7B79:~/Mystuff/Python3$

Python SSG filenames

Well my Python SSG worked really well locally, however as I previously mentiond, FTP wouldn’t copy some filenames with special characters, slashes for example, over. So I must rethink the file names.

Python Static Site Generator

I completed the main parts of generating a Computer Log Static Site. Almost all the Blog posts are created many of the images display. The Index shows all 456 of my posts. There are some image linking problems. The image link problem’s are from using WordPress’s upload feature. The images linked to my image directory work fine. I have a problem with forward slashes (which designates a directory in Linux, in the file name, for example RSTS/E so I changed those to back slashes in the file name. This worked on my Linux file system. But FTP failed to upload them on my Linux server. So I may have to rethink the filename. It took ~38 seconds to create the Static Site. All the Python code ran fast. But calling the external program to convert markdown to HTML put the brakes on the speed.

WP blog to Static Site

Started work on my own WordPress to markdown, to Static Site using Python. Got as far as creating modified markdown that includes date and title. Output filename changed to “date/time title”. Bypass external comments. Plan on using Python nodule (md-to-html) that converts markdown to html.

My own Log SSG?

Talked about it here. This whole topic started because of a WordPress search failure. I don’t think I want to devote the time right now but possibly in the future. Like I said my judoplaces static website was created in 2013, using Python/SQLite long before SSGs became a thing, before that the it was created with MS-Access. Thinking more about it, especially after recently reviewing the steps to create from WordPress. I think I could consolidate the last 5 steps into 1 step with more control including breaking up one long index page to multiple pages. Then again I could just do all those steps initially, since they already work, and then going forward, update the site using markdown, perhaps store the markdown in SQLite. Having the log in SQLite would allow me to easily retrieve the entries in date order or search on any field, rather than making Python do it. Just brainstorming right now!

Static Blog Search

The WordPress to Static Site by the Eleventy SSG went pretty well. Even considering my admitted lack of interest in web design. I made some very basic changes and improved the look considerably. The only or at least biggest thing missing is search. But there are many possibilities right off the top of my head. Perhaps the simplest is simply running a grep on the markdown files which will show the Title. Or perhaps a simple Python program that can go through the files and present a html select screen.

Static blog (11ty) to my server

After solving (so far) the FTP problem, I finally moved a test small sample of my static log file, created by 11ty to my server and it didn’t work! The problem was not using relative addressing. Or am I missing something? It seems…for a static site generator, that I should be able to simply move the files to my server and it would work. I do seem to remember reading someone’s tutorial about having to fix link addresses. Surely I’m missing something…maybe a switch? Unless 11ty is only designed to work with netlify? All the other tutorials I read seemed to use that method.

Anyhoo I wrote a python program to fix the hrefs in the main index.html. In hind site I could have simply used a text editor to change ‘href=”/posts/…’ to ‘href=”posts/…’ but at the time I thought I would have the program do more. After which…it worked. Isn’t pretty (that can be worked on), but it worked! Web design isn’t my thing but I did add some borders/white space. And here it is, maybe a few small fixes needed. No PHP…no SQL. Just static site goodness. I think I said it before but I’ll say it again, I find the fact that 11ty reads the date/time in the front matter, and sorts the index correctly (newest entries at top) very cool. Some entries such as Star Trek also include images! Today I’m approaching 500 WordPress posts and wouldn’t want to recopy everything, every time I made a change or new entry, so I can use Filezilla’s copy options to not rewrite existing entries.

So to recap, steps to create Static Site from WordPress.

  • Use WordPress export to create xml and save locally.
  • Run the blog2md-master utility to convert local xml to markdown.
  • Run my Python 3 program fixMD to change Hugo front matter to 11ty front matter.
  • Move markdown files to 11ty posts sub directory.
  • Run “npx eleventy” to create static site.
  • Fix article links in index.html to use relative addressing.
  • Move files in ‘_site’ to my website blog sub directory.