In Part 2 I described the algorithms that need to be implemented in order to transform the base matrix of an LDPC code into a reduced form that can be used by a practical encoder. As I mentioned in Part 1, we originally built a very straightforward Python implementation, where the matrix was represented literally as a bunch of rows of 0s (mostly) and 1s (rarely). Extrapolating from its performance with toy-sized codes, it would have taken months or years to reduce a life-sized (>32K bits) code. We needed something a bit faster, like a few seconds, so I set out on a C++ implementation. C++ is naturally 20-50 times faster than Python, but it would take a lot more than that.
The first, obvious, step was to change the representation to a sparse array, where only the 1 values are held explicitly. The Python code spent most of its searching arrays of 0s trying to find the occasional 1, adding a further O(n) to its execution time.
During the first phase, all the work consists of swapping rows and columns. To support this efficiently, the sparse array consists of "bitnodes" representing a 1. They are linked into lists both for the row and for the column, and contain pointers back to each of these. This means that when rows are swapped, the columns get to find out about it with no further work, and vice versa. The implementation makes extensive use of the Boost intrusive library, about which I've already eulogized. In the original implementation, the row and column lists were held in order, though I ended up rethinking this later. Here is the structure of a bitnode:
class bitnode
{
private:
typedef bi::set_member_hook<bi::link_mode<bi::auto_unlink> > row_hook_t;
typedef bi::list_member_hook<bi::link_mode<bi::auto_unlink> > col_hook_t;
row_hook_t row_hook;
col_hook_t col_hook;
row_t *my_row;
col_t *my_col;
public:
// member functions follow
.
.
};
Note the use of the intrusive member hooks, which allow the same structure to be linked into several lists (or sets). The backpointers to the row and column allow the row and column numbers to be tracked as they are swapped, which would not be the case if they were held explicitly.
This basic implementation worked well for codes with a column weight of 3, taking about 300 seconds to transform a 32K bit code. For a column weight of 5, though, which results in a much larger gap, it was unusable.
A little instrumentation showed that all the time was spent adding rows together. In the set-based implementation of sparse rows, every addition involved either the creation or the deletion of a node in a tree, a relatively expensive operation. The solution was to switch to a dense representation for the gap rows only. So, just before starting phase 2 (elimination of the ones in the F region of the matrix), the gap rows are converted to a dense representation, with one bit per possible position. This is simple enough in theory but took a lot of reworking of other structures, such as the columns. It was worth it, though: the time dropped to around 60 seconds for the column weight 3 codes, and to around 300 seconds for the column weight 5 ones.
Adding a sparse row to a dense row means walking the bitnodes in the sparse row and xor'ing the corresponding bit. Adding a dense row is just a tight loop xor'ing the 32-bit words together, an O(n) operation. These two inner loops are the key to performance - we'll come back to them later.
As always, when you speed up one part, you find another bottleneck. In this case it was phase 1 again. The best column to swap when making the diagonal was selected by simply scanning them linearly, which is obviously expensive. The solution was to keep a constantly-sorted list of the best one - actually a priority queue, implemented yet again as a boost intrusive set. However this changes constantly - when a row has been incorporated into the lower triangle, the columns it contains now have one less 1 in the region of interest. Increasing the gap also affects it. Fortunately, the row structure makes it easy to update just the columns that are directly affected, which is O(b), and then to correct their position in the list. Hence the total operation each time is O(b log(n)) which is much better than than O(n) as previously.
For a column weight of 3, this made phase 1 practically disappear as a performance concern, as I expected. But for a column weight of 5, it was still taking the majority of the time - which I didn't expect. Further analysis showed that keeping the columns in order was very expensive. Every time a row was moved to the gap, every column had to be re-sorted. On further thought, there is only one time when it helps for a column to be sorted, which is when it is being processed as the diagonal element. So just sorting it there, once per column, would work just as well and remove an O(n) element from the algorithm With this change, phase 1 moved down into the noise - for a column weight 3 code, it is about 4% of the total time.
At this point there are no further fundamental improvements to be made - the order of work to be done for each phase cannot be reduced. Further improvement can only come by coding optimizations, which will be discussed in Part 4.
Showing posts with label computers. Show all posts
Showing posts with label computers. Show all posts
Thursday, 8 December 2011
Wednesday, 7 December 2011
Algorithm Design: Efficient LDPC Encoding (Part 2: Algorithms)
In Part 1, I described the problem we are trying to solve, taking a sparse matrix and solving the corresponding system of simultaneous equations (around 33000 of them) so that we can build an efficient hardware encoder for Low Density Parity Check (LDPC) codes.
Efficient encoding requires that the original sparse matrix be transformed such that all the encoder has to do is calculate a number of parity checks. Most of these are very sparse, so they can use shared hardware. A small proportion (about 3% in a typical code) are dense, i.e. they have about the same number as 1s and 0s, and so cannot share hardware.
The resulting transformed matrix is called the "reduced" matrix, and when it is complete it has the following form:
+------------------------+--------+------------------------+
| D | E | F |
+------------------------+--------+------------------------+
| A | B | C |
+------------------------+--------+------------------------+
Rows in the D/E/F part are called the "gap" in the literature. Initially the reduced matrix is set to be identical to the base matrix, and the gap is empty. In a matrix representing a system of simultaneous equations, such as this, rows can be swapped without changing the meaning, as can columns. Also, rows can be added together (although columns cannot be). In binary addition, 1+1=0. We use these facts to rearrange the matrix into reduced form, by the following steps.
1. Transform part C into "lower triangular" form (LTF), in which everything above the main diagonal is zero. This can be done by swapping rows and columns. At each step, we look for a column that has just a single entry above the current diagonal row, then swap it with the current diagonal column. Finding a suitable column is the key to performance at this step.
2. Sometimes, we can't find such a column. This is how the gap gets created. We choose a column with the smallest number of such entries and swap that. Then we exchange rows so that the populated rows move into the gap area.
3. When this part is finished, C is in lower triangular form, but the gap is not. The next task is to complete the task for the gap, by emptying F altogether and getting E into lower triangular form. So far, all rows are still sparse, since no row or column has been changed apart from ordering.
4. For each row in F, we eliminate all bits using Gaussian elimination. Starting with the rightmost one bit, we add the corresponding row from C which has this as its rightmost bit (i.e. on the diagonal). We repeat this, moving leftward, until the F part of each row has been emptied. In the process, the rest of the row becomes dense, with on average as many 1s as 0s.
5. We now have F empty, and we need to transform E into lower triangular form. We do this by Gaussian elimination again, this time using rows from the gap. We start from the bottom and work up, creating the diagonal as we go, so that we don't put back bits that we have already eliminated.
6. Now we're done. E and C between them have a neat diagonal line with nothing above it. F is empty. A and B are still sparse, but D and the lower triangle of E are dense. All the bits in columns in A and D are data bits. The check bits, in the remainder of the matrix are generated from these.
Let's take a look at the performance of each of these steps. First we need to define some terms:
b: the number of 1 bits in a single row in the base matrix. This is small, and independent of the size of the code. It's also referred to as the row weight. We refer to the number of 1s in a single column as the column weight.
g: the number of rows in the gap region (D/E/F). Although this is much smaller than the number of data bits, it is directly linked to it. For a column weight of 3, it is about 3.3% of it. Hence anything which is O(g) is also O(n), though with a much smaller actual value.
n: the total number of rows.
The task falls into three phases:
-- Phase 1: rearrangement of rows and columns to create the C region. This has to be done once for each row (less the gap rows), and each time, we have to select the best available row. If we simply scan the rows looking for the best one, this will be O(n), making the overall task O(n2). We'll explain later how this can be made O(n log(n)). In addition we have to create the gap. Rippling a row up into the gap is O(n), and has to be done for each gap row, so the total task is O(n*g). In principle this is O(n2), but because g is so much smaller than n, with suitable design it can be kept small, comparable with the O(n log(n)) time of the row rearrangement.
-- Phase 2: eliminating all the 1s in the F region. There are g rows to deal with, and once the process starts they quickly become dense. Hence O(n*g) row additions are required, where one (the gap row) is dense, and the other, coming from the C region, is sparse. The amount of work per addition is O(b), making the whole task O(n*g*b).
-- Phase 3: eliminating the upper half of the triangle in the E region. There are g rows, and O(g) bits to be eliminated in each row, so there are O(g2) additions. Since these involve adding dense rows to each other, the amount of work per addition is O(n), making the whole task O(n*g2) - or in other words, O(n3). For small codes, this phase is dominated by phase 2, but as the code size increases it starts to dominate the total time. This is especially true if larger row or column weights are used, since the gap becomes proportionately larger (about 10% of the total rows for a column weight of 5).
These are the fundamental limits of the algorithm - no matter how clever the design, the three phases will have complexity O(n log(n)), O(n*g*b) and O(n*g2) respectively. The trick of a good implementation is to achieve these limits, and to minimize the actual values in each case. Part 3 discusses how this was done.
Efficient encoding requires that the original sparse matrix be transformed such that all the encoder has to do is calculate a number of parity checks. Most of these are very sparse, so they can use shared hardware. A small proportion (about 3% in a typical code) are dense, i.e. they have about the same number as 1s and 0s, and so cannot share hardware.
The resulting transformed matrix is called the "reduced" matrix, and when it is complete it has the following form:
+------------------------+--------+------------------------+
| D | E | F |
+------------------------+--------+------------------------+
| A | B | C |
+------------------------+--------+------------------------+
Rows in the D/E/F part are called the "gap" in the literature. Initially the reduced matrix is set to be identical to the base matrix, and the gap is empty. In a matrix representing a system of simultaneous equations, such as this, rows can be swapped without changing the meaning, as can columns. Also, rows can be added together (although columns cannot be). In binary addition, 1+1=0. We use these facts to rearrange the matrix into reduced form, by the following steps.
1. Transform part C into "lower triangular" form (LTF), in which everything above the main diagonal is zero. This can be done by swapping rows and columns. At each step, we look for a column that has just a single entry above the current diagonal row, then swap it with the current diagonal column. Finding a suitable column is the key to performance at this step.
2. Sometimes, we can't find such a column. This is how the gap gets created. We choose a column with the smallest number of such entries and swap that. Then we exchange rows so that the populated rows move into the gap area.
3. When this part is finished, C is in lower triangular form, but the gap is not. The next task is to complete the task for the gap, by emptying F altogether and getting E into lower triangular form. So far, all rows are still sparse, since no row or column has been changed apart from ordering.
4. For each row in F, we eliminate all bits using Gaussian elimination. Starting with the rightmost one bit, we add the corresponding row from C which has this as its rightmost bit (i.e. on the diagonal). We repeat this, moving leftward, until the F part of each row has been emptied. In the process, the rest of the row becomes dense, with on average as many 1s as 0s.
5. We now have F empty, and we need to transform E into lower triangular form. We do this by Gaussian elimination again, this time using rows from the gap. We start from the bottom and work up, creating the diagonal as we go, so that we don't put back bits that we have already eliminated.
6. Now we're done. E and C between them have a neat diagonal line with nothing above it. F is empty. A and B are still sparse, but D and the lower triangle of E are dense. All the bits in columns in A and D are data bits. The check bits, in the remainder of the matrix are generated from these.
Let's take a look at the performance of each of these steps. First we need to define some terms:
b: the number of 1 bits in a single row in the base matrix. This is small, and independent of the size of the code. It's also referred to as the row weight. We refer to the number of 1s in a single column as the column weight.
g: the number of rows in the gap region (D/E/F). Although this is much smaller than the number of data bits, it is directly linked to it. For a column weight of 3, it is about 3.3% of it. Hence anything which is O(g) is also O(n), though with a much smaller actual value.
n: the total number of rows.
The task falls into three phases:
-- Phase 1: rearrangement of rows and columns to create the C region. This has to be done once for each row (less the gap rows), and each time, we have to select the best available row. If we simply scan the rows looking for the best one, this will be O(n), making the overall task O(n2). We'll explain later how this can be made O(n log(n)). In addition we have to create the gap. Rippling a row up into the gap is O(n), and has to be done for each gap row, so the total task is O(n*g). In principle this is O(n2), but because g is so much smaller than n, with suitable design it can be kept small, comparable with the O(n log(n)) time of the row rearrangement.
-- Phase 2: eliminating all the 1s in the F region. There are g rows to deal with, and once the process starts they quickly become dense. Hence O(n*g) row additions are required, where one (the gap row) is dense, and the other, coming from the C region, is sparse. The amount of work per addition is O(b), making the whole task O(n*g*b).
-- Phase 3: eliminating the upper half of the triangle in the E region. There are g rows, and O(g) bits to be eliminated in each row, so there are O(g2) additions. Since these involve adding dense rows to each other, the amount of work per addition is O(n), making the whole task O(n*g2) - or in other words, O(n3). For small codes, this phase is dominated by phase 2, but as the code size increases it starts to dominate the total time. This is especially true if larger row or column weights are used, since the gap becomes proportionately larger (about 10% of the total rows for a column weight of 5).
These are the fundamental limits of the algorithm - no matter how clever the design, the three phases will have complexity O(n log(n)), O(n*g*b) and O(n*g2) respectively. The trick of a good implementation is to achieve these limits, and to minimize the actual values in each case. Part 3 discusses how this was done.
Tuesday, 6 December 2011
Algorithm Design: Efficient LDPC Encoding (Part 1: Background)
I've been working lately on a system design which requires, among other things, highly effective and efficient error-correcting codes (ECC). We've decided to use a Low Density Parity Check (LDPC) code. These are currently considered to be the best "soft" ECCs, i.e. where there is information about the reliability of each received bit as well as its putative value. The story behind LDPCs is interesting: they were invented by Robert Gallager in his PhD thesis in 1960, but they were way beyond contemporary computing power. It didn't help that when he wrote the definitive textbook on ECCs in 1966, he didn't mention them! So they languished, forgotten, until a decade ago. By then TurboCodes had been independently invented. They also provided a means for "near Shannon limit coding", i.e. extracting as much data from a noisy signal as theoretically possible.
LDPCs have two properties which led to the problem I needed to solve. First, there is no formula that provides the best code for a given set of constraints (block size and code rate). You can use the same general scheme to build ten different codes, the details being decided by a random number generator, and some will be significantly better than others. That means that to find the code you want to use in practice, you need to generate a whole bunch of them and try them out over a large number of messages and error densities.
That leads to the second problem. An LDPC starts out as a very sparse matrix, describing a large number of parity checks each of which covers a small number of bits - hence the name. We want to have 32768 bits of user data, and a reasonable configuration is to have each bit covered by three checks. If we use a half-rate code (same number of data bits and check bits) then each check covers six bits. So we have a matrix where each row is 64K bits long and has just six 1 bits.
The matrix doesn't say anything about which bits are data and which are check bits, only that a valid codeword has to satisfy all the checks. So given 32K data bits, the way to generate the corresponding 32K check bits is to solve the 32K simultaneous equations that the sparse matrix implicitly describes. Easy!
Well, no, not easy at all. The practical use of LDPCs requires a transformation of the matrix into something that normal hardware or software can encode in a linear and reasonable amount of time. Solving the equations directly is an O(n3) problem, i.e. the time required increases with the cube of the number of unknowns. So we have to some preprocessing on the matrix to get it into a form that the hardware can work with. There's an excellent paper by Qi and Goertz describing how to go about this. The algorithm it describes is, not surprisingly, also O(n3). This needs to be run for every trial code, and we would like to try hundreds of them.
Our first attempt at coding the algorithm was written in Python, using the obvious data representation, i.e. a big matrix containing mostly 0s and a few 1s. It was written so we could understand the algorithms and piece together a complete system, rather than for performance. On a "toy" code of a few hundred bits, it took a couple of minutes to run. On slightly larger codes - nowhere near the size we need for our system - it took most of the day. By extrapolation, to generate a life-size code would have taken months or years.
Clearly, we needed an implementation more focused on performance - not just code optimization, but selecting algorithms to minimize the time at step of the algorithm. And that is where it begins to get interesting. More on that in Part 2.
LDPCs have two properties which led to the problem I needed to solve. First, there is no formula that provides the best code for a given set of constraints (block size and code rate). You can use the same general scheme to build ten different codes, the details being decided by a random number generator, and some will be significantly better than others. That means that to find the code you want to use in practice, you need to generate a whole bunch of them and try them out over a large number of messages and error densities.
That leads to the second problem. An LDPC starts out as a very sparse matrix, describing a large number of parity checks each of which covers a small number of bits - hence the name. We want to have 32768 bits of user data, and a reasonable configuration is to have each bit covered by three checks. If we use a half-rate code (same number of data bits and check bits) then each check covers six bits. So we have a matrix where each row is 64K bits long and has just six 1 bits.
The matrix doesn't say anything about which bits are data and which are check bits, only that a valid codeword has to satisfy all the checks. So given 32K data bits, the way to generate the corresponding 32K check bits is to solve the 32K simultaneous equations that the sparse matrix implicitly describes. Easy!
Well, no, not easy at all. The practical use of LDPCs requires a transformation of the matrix into something that normal hardware or software can encode in a linear and reasonable amount of time. Solving the equations directly is an O(n3) problem, i.e. the time required increases with the cube of the number of unknowns. So we have to some preprocessing on the matrix to get it into a form that the hardware can work with. There's an excellent paper by Qi and Goertz describing how to go about this. The algorithm it describes is, not surprisingly, also O(n3). This needs to be run for every trial code, and we would like to try hundreds of them.
Our first attempt at coding the algorithm was written in Python, using the obvious data representation, i.e. a big matrix containing mostly 0s and a few 1s. It was written so we could understand the algorithms and piece together a complete system, rather than for performance. On a "toy" code of a few hundred bits, it took a couple of minutes to run. On slightly larger codes - nowhere near the size we need for our system - it took most of the day. By extrapolation, to generate a life-size code would have taken months or years.
Clearly, we needed an implementation more focused on performance - not just code optimization, but selecting algorithms to minimize the time at step of the algorithm. And that is where it begins to get interesting. More on that in Part 2.
Friday, 9 September 2011
Dedicated Wallpaper Screen - everyone should have one!
One which I can really recommend is to collect all of your favorite photos, and have a second screen on your desk which shows you a randomly changing selection of them. They don't have to be pictures that you've taken yourself, of course, but there's something especially cheering about seeing places you've been, cool things you've done, happy times you've had...
It occurred to me a while ago that you could easily hook up a screen like this, to a superannuated computer. Gee, you could even build a little desktop gadget with a screen and a microprocessor... then of course before I could do anything about it, Philips brought one out. They were rapidly followed by a bunch of no-name Chinese products with an extra twist - they contained a virus that infected your computer as soon as you connected them. Neat trick. I had an actual Philips one, which was malware-free but not explosion-free. One day I noticed that it was a funny shape, and when I took it apart the internal lithium-ion battery had exploded, with enough force to bend the solid metal frame. It has never been quite the same since, and anyway there's no longer room for it on my desk.
At the same time I bought the picture frame, I started assembling my favourite pictures - the same ones as in my Flickr account, with the addition of some family pictures that I don't particularly want to share with the world at large. I found a wallpaper changer that worked well for me, and whenever my computer was idle, they'd scroll by in front of me. It was great, but there's one problem: if I'm not actually using the machine, I'm probably not looking at it. And if I am using it, of course, the wallpaper is invisible under the clutter of a dozen windows.
Fast forward to my new Linux system. At the same time I bought it, I also bought a new monitor. There was nothing wrong with the old one, except a broken stand elegantly patched up with cable ties, but the new one has more pixels, and bigger is always better (n'est-ce pas?). So, I suddenly had a spare large format (1600 x 1200) monitor, and thanks to a superhuman feat of tidying, a space for it on my desk.
It took me a while to get round to making this all work, though. There are people who have had multi-monitor systems for years, but I've always preferred to have a single large one - hence the present 1920 x 1200 display.
It seemed obvious to me that I would need a second graphics card, so off I went to Fry's and bought a low-end one, a Realtek HD5450. I've got used to how well Linux deals with new hardware, so I just plugged it in and expected it to work. My optimism was misplaced, however. I'm running a recent version of Linux (11.04, Natty) and all the bits weren't in the right place, fixed with a bit of googling. No matter what I did, though, the system only used the built-in graphics on the motherboard.
A bit more googling showed me how to change the BIOS setup so it would use the new card - but then it would only use the new card. There was no way to get it to use both of them at once.
There was a seriously heart-stopping moment in the middle of all this. There was one BIOS setting that resulted in a psychedelic display as the system booted - a patchwork of constantly changing colors. Eventually the system came up normally - but how could I change the BIOS settings again? It seemed for a short while as if the only thing to do would be to buy a new motherboard! Luckily, unplugging the new card magically made everything work again.
In the end the solution was simple. You can run two monitors off the same graphics card, just plugging them into two different sockets. And I didn't even need the new card - thank you Fry's for an extremely liberal return policy.
It still wasn't completely over, though. Convincing Linux to run both displays wasn't at all obvious. Not only did I have to tell both Gnome (the desktop system) and the monitor drivers, but I had to do it in the right order - otherwise one undid the changes to the other. The X-windows configuration had to be changed manually, using the magic command "sudo dpkg-reconfigure xserver-xorg". Somehow this reads your mind, figures out what you're trying to do, then generates the corresponding xorg.conf file.
Finally I had it all working. Still not there, though. The "wallpaper" monitor is to the left on my desk. But Gnome wants to put everything on the left screen, unless you've explicitly moved it to the other one. There's a box you can click to make a different screen primary, but it only has a limited effect. Luckily, most X apps remember where you last put them, so it's just a question of "training" the apps you use most. Every now and then, though, I click on something, and can't understand why nothing has happened. Then I look at the wallpaper screen, and it's over there. So I call it, "here Fido", over it comes, and another app has got it figured. (Well, actually I drag it with the mouse).
And what a constant pleasure it is, to see all these pictures of things I've done and places I've been. While I've been typing this, I've had...
- the tiny landing strip in the remotest part of Baja California, whwere we went whale spotting
- a beautiful, moss-clad waterfall in the Oirase Gorge in northern Japan
- a still-life composition of freshly caught fish in the market in Cap Breton, France
- several shots of the Golden Gate Bridge, taken from different aircraft at different times, including the ones taken while flying under it in the heli
- a visit to Potsdam, Germany in 1986, when it was still East Germany, with just a single smelly Trabant visible in the whole of a huge plaza
Friday, 26 August 2011
Boost: a retrospective (part 3) - the Bad and the Ugly
In part 2 I talked about my favorite elements of the Boost libraries. Boost is wonderful, but even so there are things that are not so good. These, the ones which (in my opinion) are best avoided, form the subject of this post.
Serialization
I wrote a while ago about my frustration with this library. It seemed the perfect solution to a data pickling need I had, until I discovered that it can't cope with polymorphism. It claims to, but it randomly crashes deeply nested in incomprehensible function calls if you try. There may have been a solution, but life is just too short to figure it out. The reason for all this is that its authors decided to invent their very own subclassing scheme, completely orthogonal to the one that C++ uses. They may have had their reasons, but it's a complex subject and clearly they missed something.
Asio
If you've ever needed to do low-level socket I/O, you've probably been tempted to write an object wrapper around the C function calls and data structures. You may even have taken a look at Boost to see if they have already done this. In which case, you'll find that they have. I've certainly been down this path, and discovered Boost Asio at the end of it.
You will next discover that Asio is extremely complex, with all kinds of interacting classes that you have to be aware of and create. I spent a day or so trying to get my head around it, finally getting to the point where I felt safe putting fingers to keyboard. Then I discovered that despite all that complexity, it couldn't do what I needed. This was nothing fancy, just listen on a port, and create a thread to handle each TCP session as it arrives. Turns out Asio has a race condition - by design - which can result in missed connections. Some searching showed that there's a workaround for this, but it's complex and requires even more delving into its complexities - and isn't without its own problems anyway.
I had a long meeting to attend, so I figured I'd print the documentation and peruse it during the meeting. Over 800 pages later, my meeting had finished anyway, but the printer still hadn't. At this point, I decided that anything which takes 800 pages to describe it - for such a relatively simple function, this isn't Mathematica after all (1465 pages) - just can't be worth the learning curve.
I wrote my own family of socket classes. It actually took me less time to write and debug than it did to print the Asio documentation, never mind read it! I've been very happily using them ever since. Probably, you will do the same, but if you'd like to use mine, you're welcome. You can find them here.
The Build System
Everyone knows Make. It's convoluted, nearly incomprehensible, and a syntactic nightmare, but everyone has used it and can bodge their way out of a tight corner if they need to.
But why use something everyone knows, when you can invent something unique of your own? Sadly, this is the path that Boost took. They have their own unique build system called Bjam. I'm sure it's very elegant compared to Make - it would take a huge effort not to be - but it's still very complex, and poorly documented too. In fairness, it does (mostly) "just work" if you need to build Boost from sources. But if for whatever reason you do need to get under the covers, woe betide you.
I discovered this when I needed to cross-build Boost for our embedded processor. This is always tricky because of the config stage, where the build system looks to see what capabilities the system has, where things are located and so on. For a cross-build, of course, you can't auto-discover this just by poking around at the system you're running on. That part went OK, though. However editing the build files to pick up the right cross-compiler, cross-linker and so on, was just impossible. I found quite a bit about it on the web, but never quite enough to make it work.
Fortunately, our hardware ran a complete Linux system and with a little fiddling we could just build it native on our box. But if you can't do this - and most embedded systems can't - then you can forget using Boost. Which is a shame.
Serialization
I wrote a while ago about my frustration with this library. It seemed the perfect solution to a data pickling need I had, until I discovered that it can't cope with polymorphism. It claims to, but it randomly crashes deeply nested in incomprehensible function calls if you try. There may have been a solution, but life is just too short to figure it out. The reason for all this is that its authors decided to invent their very own subclassing scheme, completely orthogonal to the one that C++ uses. They may have had their reasons, but it's a complex subject and clearly they missed something.
Asio
If you've ever needed to do low-level socket I/O, you've probably been tempted to write an object wrapper around the C function calls and data structures. You may even have taken a look at Boost to see if they have already done this. In which case, you'll find that they have. I've certainly been down this path, and discovered Boost Asio at the end of it.
You will next discover that Asio is extremely complex, with all kinds of interacting classes that you have to be aware of and create. I spent a day or so trying to get my head around it, finally getting to the point where I felt safe putting fingers to keyboard. Then I discovered that despite all that complexity, it couldn't do what I needed. This was nothing fancy, just listen on a port, and create a thread to handle each TCP session as it arrives. Turns out Asio has a race condition - by design - which can result in missed connections. Some searching showed that there's a workaround for this, but it's complex and requires even more delving into its complexities - and isn't without its own problems anyway.
I had a long meeting to attend, so I figured I'd print the documentation and peruse it during the meeting. Over 800 pages later, my meeting had finished anyway, but the printer still hadn't. At this point, I decided that anything which takes 800 pages to describe it - for such a relatively simple function, this isn't Mathematica after all (1465 pages) - just can't be worth the learning curve.
I wrote my own family of socket classes. It actually took me less time to write and debug than it did to print the Asio documentation, never mind read it! I've been very happily using them ever since. Probably, you will do the same, but if you'd like to use mine, you're welcome. You can find them here.
The Build System
Everyone knows Make. It's convoluted, nearly incomprehensible, and a syntactic nightmare, but everyone has used it and can bodge their way out of a tight corner if they need to.
But why use something everyone knows, when you can invent something unique of your own? Sadly, this is the path that Boost took. They have their own unique build system called Bjam. I'm sure it's very elegant compared to Make - it would take a huge effort not to be - but it's still very complex, and poorly documented too. In fairness, it does (mostly) "just work" if you need to build Boost from sources. But if for whatever reason you do need to get under the covers, woe betide you.
I discovered this when I needed to cross-build Boost for our embedded processor. This is always tricky because of the config stage, where the build system looks to see what capabilities the system has, where things are located and so on. For a cross-build, of course, you can't auto-discover this just by poking around at the system you're running on. That part went OK, though. However editing the build files to pick up the right cross-compiler, cross-linker and so on, was just impossible. I found quite a bit about it on the web, but never quite enough to make it work.
Fortunately, our hardware ran a complete Linux system and with a little fiddling we could just build it native on our box. But if you can't do this - and most embedded systems can't - then you can forget using Boost. Which is a shame.
Tuesday, 23 August 2011
VirtualBox - virtually complete: part 1
My new Linux machine is almost complete now. It has been running for a couple of months. Since I lost my laptop along with the company I worked for, and haven't seen a reason to buy another one yet, the Linux machine has become my main computing platform for just about everything.
However there are some things which can only be done on Windows. One is iTunes - I only use it as a backup for my iPhone, but for that it is indispensable. Another is the package that updates the navigation data for the plane. Then there is expensive software I bought for Windows and don't plan to buy again - Mathematica, CorelDraw and so on. Not to mention my HP printer/scanner which has no Linux driver. So I've had two machines sitting next to each other, with a KVM switch to go back and forth when needed.
Of course the solution to this is obvious - virtual machines. I've been idly looking at VirtualBox, one of the open-source VM solutions, for a while. It took something odd to make me dig a bit deeper - the discovery that our electricity supplier (PG&E) has a web page where you can see your hourly consumption. It's fascinating to study, and one thing it showed is that my two desktop computers were accounting for about 60% of the house's background electricity usage. So, turn one of them off, instant 30% energy saving. (If only I could do something equally miraculous for the pool, which accounts for over half of our electricity).
I started by creating a second Linux system. That seemed less scary, though as it turned out it was harder than Windows. It's simple enough - tell VBox to create a virtual machine, then boot it with a Ubuntu CD in the drive and it pretty much goes all by itself. My plan for this VM was to use it as a web server for stuff I want to host at home, rather than through my web provider. It all went well, until I tried to set up a "shared folder".
In its pure form, the VM concept means that the VM runs in complete isolation in a bubble inside the real operating system. This is fine until you do actually want to share stuff. You can do it over the net, using FTP or whatever, but even that isn't "out of the box". By default, Vbox creates VMs using internal NAT addresses, so there is no way to access the VM from anywhere else, including the host. That can be fixed with a couple of clicks, selecting "Bridged Adapter" in the "Attachedto" drop-down, instead of "NAT". But still, it's a clunky way to do things.
So you can also create a "shared folder". This is just a regular directory on the host system, but it looks like a remote filesystem to the guest. It's easy enough to set up, but I just could not get it to work. I successfully mounted it just once. After that, attempts to mount it always failed with "no such device" (or some such). Even deleting it and creating a new one didn't work.
Finally I discovered that, because I'd ticked the "auto-mount" box, it was being automatically mounted in /media. Well, duh, I suppose.
With that done, my Linux VM was usable. Fortunately, because it turns out in my main (host) system, Firefox had autoupdated itself to V6 - and the Flash player doesn't work an more. No more stock charts from Google, no more news video clips from the BBC. This seems to be a problem that a handful of people have run into, with no solution. So for now, I just use an older version of Firefox running inside the Linux VM whenever I need Flash.
The next step was to install a Windows VM. But this is already too long, so that will be for another time
However there are some things which can only be done on Windows. One is iTunes - I only use it as a backup for my iPhone, but for that it is indispensable. Another is the package that updates the navigation data for the plane. Then there is expensive software I bought for Windows and don't plan to buy again - Mathematica, CorelDraw and so on. Not to mention my HP printer/scanner which has no Linux driver. So I've had two machines sitting next to each other, with a KVM switch to go back and forth when needed.
Of course the solution to this is obvious - virtual machines. I've been idly looking at VirtualBox, one of the open-source VM solutions, for a while. It took something odd to make me dig a bit deeper - the discovery that our electricity supplier (PG&E) has a web page where you can see your hourly consumption. It's fascinating to study, and one thing it showed is that my two desktop computers were accounting for about 60% of the house's background electricity usage. So, turn one of them off, instant 30% energy saving. (If only I could do something equally miraculous for the pool, which accounts for over half of our electricity).
I started by creating a second Linux system. That seemed less scary, though as it turned out it was harder than Windows. It's simple enough - tell VBox to create a virtual machine, then boot it with a Ubuntu CD in the drive and it pretty much goes all by itself. My plan for this VM was to use it as a web server for stuff I want to host at home, rather than through my web provider. It all went well, until I tried to set up a "shared folder".
In its pure form, the VM concept means that the VM runs in complete isolation in a bubble inside the real operating system. This is fine until you do actually want to share stuff. You can do it over the net, using FTP or whatever, but even that isn't "out of the box". By default, Vbox creates VMs using internal NAT addresses, so there is no way to access the VM from anywhere else, including the host. That can be fixed with a couple of clicks, selecting "Bridged Adapter" in the "Attachedto" drop-down, instead of "NAT". But still, it's a clunky way to do things.
So you can also create a "shared folder". This is just a regular directory on the host system, but it looks like a remote filesystem to the guest. It's easy enough to set up, but I just could not get it to work. I successfully mounted it just once. After that, attempts to mount it always failed with "no such device" (or some such). Even deleting it and creating a new one didn't work.
Finally I discovered that, because I'd ticked the "auto-mount" box, it was being automatically mounted in /media. Well, duh, I suppose.
With that done, my Linux VM was usable. Fortunately, because it turns out in my main (host) system, Firefox had autoupdated itself to V6 - and the Flash player doesn't work an more. No more stock charts from Google, no more news video clips from the BBC. This seems to be a problem that a handful of people have run into, with no solution. So for now, I just use an older version of Firefox running inside the Linux VM whenever I need Flash.
The next step was to install a Windows VM. But this is already too long, so that will be for another time
Tuesday, 9 August 2011
Python and Tkinter: wonderful
Having time on my hands at the moment, and no work commitments - that's another story though - I decided to start taking a new look at the robotics stuff I was playing with a year or so ago.
I'd written nearly all of the code to make a six-legged robot - a hexapod - walk with various different gaits and postures - the so-called inverse kinematics. It was in straight C, since I intended it run on a little embedded CPU which had no support for C++, nor floating point for that matter. And I'd built a development environment, including visualisation for the leg movements, using Visual Studio.
Things have moved on since then, though. For one thing I've pretty much switched to Linux for my computing environment. For another, the Roboard has really become the obvious onboard computer - it is now available with Linux, and it offers a full-function x86 including floating point, in a tiny size that will fit in my fairly small hexapod. And since it supports full GCC, I can write the code in C++. The C code is just so cluttered - I can't for the life of me imagine why anyone would prefer to code in C. It's full of irrelevant details that make it hard to read and even harder to get it to work. So, a rewrite is called for.
The only problem, is the GUI that I'd painfully created using the Visual Studio tools. Painful because there are 6 legs, and each has numerous parameters and state variables. I'd created the dialog box from hell. Every tiny change meant nudging numerous components around to get it to look right. What a pain. But it was done, for now anyway.
That was when I thought about Tkinter, which I've never used before. I've become a huge fan of Python in the last couple of years, using it for anything where performance is not a big deal. I also wrote a very powerful Python-based scripting system for my now-defunct employer, using Boost Python. So using Python and Tkinter for the GUI was kind of an obvious thing to do.
Somewhere in the mists of history I acquired Python and Tkinter Programming, which I think is the definitive book on the topic. I skimmed that, and with frequent help from Google - especially this site - started putting my new GUI together.
What a pleasure! Tkinter automatically takes care of making a reasonable layout, given some general guidance through the pack and grid functions. You no longer have to think about the minutiae of positioning, or spend ages getting boxes to line up with each other. I just couldn't help putting together a bit of infrastructure for collections of config variables, so they are now super-easy - just a list of names and default values and Python and Tkinter take care of everything.
In total it has probably taken me about 6 hours to get everything together - but that included learning Tkinter from scratch and writing quite a bit of infrastructure. And now I have everything I need to control my inverse kinematics, and have an animated visualisation of what it's doing.
I'll never do GUIs any other way now. Tkinter is wonderful!
I'd written nearly all of the code to make a six-legged robot - a hexapod - walk with various different gaits and postures - the so-called inverse kinematics. It was in straight C, since I intended it run on a little embedded CPU which had no support for C++, nor floating point for that matter. And I'd built a development environment, including visualisation for the leg movements, using Visual Studio.
Things have moved on since then, though. For one thing I've pretty much switched to Linux for my computing environment. For another, the Roboard has really become the obvious onboard computer - it is now available with Linux, and it offers a full-function x86 including floating point, in a tiny size that will fit in my fairly small hexapod. And since it supports full GCC, I can write the code in C++. The C code is just so cluttered - I can't for the life of me imagine why anyone would prefer to code in C. It's full of irrelevant details that make it hard to read and even harder to get it to work. So, a rewrite is called for.
The only problem, is the GUI that I'd painfully created using the Visual Studio tools. Painful because there are 6 legs, and each has numerous parameters and state variables. I'd created the dialog box from hell. Every tiny change meant nudging numerous components around to get it to look right. What a pain. But it was done, for now anyway.
That was when I thought about Tkinter, which I've never used before. I've become a huge fan of Python in the last couple of years, using it for anything where performance is not a big deal. I also wrote a very powerful Python-based scripting system for my now-defunct employer, using Boost Python. So using Python and Tkinter for the GUI was kind of an obvious thing to do.
Somewhere in the mists of history I acquired Python and Tkinter Programming, which I think is the definitive book on the topic. I skimmed that, and with frequent help from Google - especially this site - started putting my new GUI together.
What a pleasure! Tkinter automatically takes care of making a reasonable layout, given some general guidance through the pack and grid functions. You no longer have to think about the minutiae of positioning, or spend ages getting boxes to line up with each other. I just couldn't help putting together a bit of infrastructure for collections of config variables, so they are now super-easy - just a list of names and default values and Python and Tkinter take care of everything.
In total it has probably taken me about 6 hours to get everything together - but that included learning Tkinter from scratch and writing quite a bit of infrastructure. And now I have everything I need to control my inverse kinematics, and have an animated visualisation of what it's doing.
I'll never do GUIs any other way now. Tkinter is wonderful!
Friday, 20 May 2011
My Very First Linux System
Ubuntu rocks. For more, read on...
About a year ago, I converted my home desktop machine from Windows XP to Windows 7, hoping to improve its dire performance problems. That was not a success. Not only was the Windows 7 performance if anything worse than XP, but a number of other things stopped working as well. Based on my experience I cannot imagine why anyone would ever choose to run W7 instead of XP. W7 is full of poorly conceived "security" misfeatures, making it painful to use and even harder to manage. For example, it's impossible to save an image from a webpage to local storage. And I never did get the Cisco VPN to work.
I've been thinking for a while that it would be a good idea to have a Linux server at home. I've switched just about all my non-embedded stuff at work from Windows and Visual Studio to run on Linux systems. VS is actually quite decent but things need to end up on Linux anyway and keeping them dual-platform is too much trouble.
When I first used Unix, Linux hadn't even been thought of. I equipped my network architecture team at DEC with workstations in about 1988. Of course at DEC we didn't have the System Whose Name Must Remain Unspoken, we had Ultrix - which was just BSD Unix. And Linux is both our development and our production environment at work. Running Linux as a user is no issue, but I was worried about the sys admin aspect of the thing. My perception of Linux was of constantly having to build things from source, of irrenconcilable version conflicts and unfindable drivers and bits of operating system. So I kept putting it off.
Then a friend told me about a system they had just bought. Six-core 3.2 GHz AMD CPU, 8 GBytes of memory... all for under $500. That sounded too good to be true, but I went to the Tiger Direct web site and there it was. The temptation was too strong. It's difficult to buy a PC-type system without getting - and paying for - Windows, whether you want it or not. NewEgg for example won't sell you a barebones system like this.
A few days later it all showed up in several large packages. I've always bought assembled systems before, but this one came as a collection of parts - case, power supply, motherboard, CPU, disk, memory, and a few other bits and pieces. I was apprehensive about putting it all together but in fact it was easy. The only trickly part was getting the cheap-and-nasty metal plate around all the motherboard conneectors to fit propely. Functionally speaking, this isn't strictly necessary, but it would look funny without it.
Actually the system came without memory, because it was backordered, and also I'd ordered the wrong part for the CD/DVD drive as well. So it took a couple of trips to Fry's to get everything required. I'd also bought an Ubuntu book which conveniently had a CD in the back, avoiding the problem of downloading the distro and burning a CD - for which I'm not sure I have the software.
And then came the miraculous part. I put the CD in the drive and booted the machine. Ubuntu came up, and with a couple of clicks started to install itself. Nothing could ever have been easier! Once my new system was running, there were numerous things I wanted to install. For every single one of them, a single command and a short wait downloaded it, installed it and got it running. No scratching round trying to find ancient CDs, no complicated licence agreements, no complex configuration screens. And better yet, no Windows registry! Everything just installed and run.
I'd never used Gnome or any other Linux GUI before. Up to now, all my Linux usage has been from the command line - which I rather like, actually. I had low expectations of Gnome but I couldn't have been more wrong. It does everything Windows does, and without all the irritating bogus security misfeatures. I was especially pleasantly surprised by OpenOffice, which can read and edit all my Word, Excel and Powerpoint files. And it's free.
There was only one thing that gave me problems. No surprise, it was the interconnection with my Windows system which, for now anyway, I have kept in place. There is a piece of software, Samba, that allows a Linux system to provide a Windows file server. Unfortunately it has to accomodate all of the superfluous complexity and options of Windows, so configuring it is a nightmare and I just couldn't get it to work. Finally a friend pointed me to SWAT which straightened it all out for me. A key item is that the accounts and passwords for Samba have absolutely nothing to do with any other accounts and passwords on the system. Connection in the other direction, accessing Windows files from my Linux system, is still distinctly shaky. It kind of works, sometimes, but keeps running into all those security misfeatures that made me dislike W7 so much in the first place. I have yet to get it so the Linux machine can write files on Windows, even using the adminsitrator account.
One thing I wanted to do was open up the machine for very limited outside access, so I can retrieve my files from it. This was amazingly easy - install Apache and it just works. Then a little configuration on my Linksys router, five minutes to create an account with dyndns.org, and now I can (with a password of course) access my files from anywhere.
I love my new Linux machine. Next step: move Windows to be a VM under Linux. More on that later...
About a year ago, I converted my home desktop machine from Windows XP to Windows 7, hoping to improve its dire performance problems. That was not a success. Not only was the Windows 7 performance if anything worse than XP, but a number of other things stopped working as well. Based on my experience I cannot imagine why anyone would ever choose to run W7 instead of XP. W7 is full of poorly conceived "security" misfeatures, making it painful to use and even harder to manage. For example, it's impossible to save an image from a webpage to local storage. And I never did get the Cisco VPN to work.
I've been thinking for a while that it would be a good idea to have a Linux server at home. I've switched just about all my non-embedded stuff at work from Windows and Visual Studio to run on Linux systems. VS is actually quite decent but things need to end up on Linux anyway and keeping them dual-platform is too much trouble.
When I first used Unix, Linux hadn't even been thought of. I equipped my network architecture team at DEC with workstations in about 1988. Of course at DEC we didn't have the System Whose Name Must Remain Unspoken, we had Ultrix - which was just BSD Unix. And Linux is both our development and our production environment at work. Running Linux as a user is no issue, but I was worried about the sys admin aspect of the thing. My perception of Linux was of constantly having to build things from source, of irrenconcilable version conflicts and unfindable drivers and bits of operating system. So I kept putting it off.
Then a friend told me about a system they had just bought. Six-core 3.2 GHz AMD CPU, 8 GBytes of memory... all for under $500. That sounded too good to be true, but I went to the Tiger Direct web site and there it was. The temptation was too strong. It's difficult to buy a PC-type system without getting - and paying for - Windows, whether you want it or not. NewEgg for example won't sell you a barebones system like this.
A few days later it all showed up in several large packages. I've always bought assembled systems before, but this one came as a collection of parts - case, power supply, motherboard, CPU, disk, memory, and a few other bits and pieces. I was apprehensive about putting it all together but in fact it was easy. The only trickly part was getting the cheap-and-nasty metal plate around all the motherboard conneectors to fit propely. Functionally speaking, this isn't strictly necessary, but it would look funny without it.
Actually the system came without memory, because it was backordered, and also I'd ordered the wrong part for the CD/DVD drive as well. So it took a couple of trips to Fry's to get everything required. I'd also bought an Ubuntu book which conveniently had a CD in the back, avoiding the problem of downloading the distro and burning a CD - for which I'm not sure I have the software.
And then came the miraculous part. I put the CD in the drive and booted the machine. Ubuntu came up, and with a couple of clicks started to install itself. Nothing could ever have been easier! Once my new system was running, there were numerous things I wanted to install. For every single one of them, a single command and a short wait downloaded it, installed it and got it running. No scratching round trying to find ancient CDs, no complicated licence agreements, no complex configuration screens. And better yet, no Windows registry! Everything just installed and run.
I'd never used Gnome or any other Linux GUI before. Up to now, all my Linux usage has been from the command line - which I rather like, actually. I had low expectations of Gnome but I couldn't have been more wrong. It does everything Windows does, and without all the irritating bogus security misfeatures. I was especially pleasantly surprised by OpenOffice, which can read and edit all my Word, Excel and Powerpoint files. And it's free.
There was only one thing that gave me problems. No surprise, it was the interconnection with my Windows system which, for now anyway, I have kept in place. There is a piece of software, Samba, that allows a Linux system to provide a Windows file server. Unfortunately it has to accomodate all of the superfluous complexity and options of Windows, so configuring it is a nightmare and I just couldn't get it to work. Finally a friend pointed me to SWAT which straightened it all out for me. A key item is that the accounts and passwords for Samba have absolutely nothing to do with any other accounts and passwords on the system. Connection in the other direction, accessing Windows files from my Linux system, is still distinctly shaky. It kind of works, sometimes, but keeps running into all those security misfeatures that made me dislike W7 so much in the first place. I have yet to get it so the Linux machine can write files on Windows, even using the adminsitrator account.
One thing I wanted to do was open up the machine for very limited outside access, so I can retrieve my files from it. This was amazingly easy - install Apache and it just works. Then a little configuration on my Linksys router, five minutes to create an account with dyndns.org, and now I can (with a password of course) access my files from anywhere.
I love my new Linux machine. Next step: move Windows to be a VM under Linux. More on that later...
Thursday, 11 February 2010
The Upgrade: Help! I can't get help!
Now that my VPN problem is solved, my Windows 7 system seems to be working reasonably well. I still can't find the IE cache, or rather I can find it and some things are there but a lot of other things aren't. And videos always start jerkily in the first few seconds, but then they get smoother. All of this is survivable.
But I did come across yet another "what were they thinking" a few days ago. I have quite a lot of old software on the machine - my very own Winlife32, dating back in its origins to 1992, for one. I have a version of "Corel Essentials" that I bought nearly ten years ago. I don't use it enough to justify going and spending $100 on a newer version, but it's handy when I do need it. PhotoPaint is an OK-ish inferior version of PhotoShop, with the great advantage compared to PhotoShop that I know how to use it. And CorelDraw is absolutely indispensible for making jam labels. So the other day I needed to annotate the pictures for my Divisumma Diary. I wanted to draw some arrows on them, which ought not to be difficult to do, but I couldn't find the trick. For once Google was no help, so I was driven to the ultimate recourse, the documentation. Or rather, since there isn't any, the online help. So I clicked the help button. And my brand new Windows 7 system said "I don't know how to open this file", or words to that effect. It turns out that brand spanking new Windows 7 can't open those crufty old fashioned .HLP files that came out with Windows 3.1. Which is a major pain if you have applications that still work that way, like my Corel stuff or indeed Winlife32.
Ah! But then there's an option that says "would you like to download WINHLP.EXE for Windows 7?" So I said yes, and was led through a maze of twisty little web pages, until I got asked which of several files I wanted to download. There was no basis for choosing, and the one that looked most likely (it had 32 in the name where the others had 64) downloaded then said, "this program will not run on your system". Eventually random selection found the right one. And then Corel help worked fine. So does WInlife32 help, although I have now discovered that the API it uses for drawing doesn't work on Windows 7 or Vista, so I guess I'll have to figure out what gratuitous change Microsoft has made there and figure out the fix for that.
So, some genius in Redmond decided that although there's a perfectly fine version of Winhelp that runs on Windows 7, and there are undoubtedly thousands of old apps out there that want to use it, they wouldn't put it on the (huge and bloated) distribution. Instead you have to go through an arcane process to download it, whereupon it works fine. What were they thinking???
Oh, and it turns out that there is no way to draw arrows with PhotoPaint, at least not the version I have, short of drawing the heads by hand, line by line. By the way there's also no way to draw a simple oval line with transparent fill, such as you might put around something to draw attention to it. I guess maybe I really should get one of the cheaper flavors of PhotoShop.
Monday, 18 January 2010
On Upgrading - or "what were they thinking?"
My desktop machine at home has had creeping software sickness for a long time. It was new about 4 years ago and has gradually got less and less likely to succeed at any given task. Visual Studio crashes quite often, which I've never seen anywhere else. Google Earth kind-of runs but is pretty much unusable. And so on. I've no idea why these things happen, there's no logical reason why software should wear out, but it seems that it does.
I resisted Vista for all of its sorry lifetime, wisely as it turned out. But everyone seems to be saying nice things about Windows 7, so I decided it was time to upgrade. At the same time I decided to swap the 250 Gbyte hard drive for a brand new 2 terabyte one. Not that I have anything that will use up that much disk space, but 250 GB was getting full.
The machine is something called an HP Media Center. It was a distress purchase, my previous machine had scrambled its hard drive and I had exactly 10 hours to get something important done, so I just rushed out to Fry's and bought what they recommended. Until now, I'd never had any reason to take the covers off. When I did, I found that the hard drive was totally, utterly inaccessible, hidden inside the machine, and mounted vertically (rather than horizontally as usual). No normal screwdriver could access any of the mounting screws. I tried undoing various other things, but nothing gave access to the screws. Eventually I figured I could just put the new drive loose on the floor of the box. But, since I don't give up easily, I wondered how the factory had ever managed to assemble the machine. More on that later.
So, having swapped the drive cables over to my brand-new, unformatted 2 TB disk, it was time to install Windows 7, which I had (legitimately) on a USB memory stick. (As an aside, in about 1994 I worked on one of the very first video-on-demand projects, at DEC. Part of the system was a 1 TB drive farm, which occupied four full-size racks. Of course the whole project was more than a decade ahead of its time, and flopped completely).
Since I had a new disk, I actually wanted to do an install from scratch. But still there is a great mystery here. Microsoft built Windows 7 because the adoption rate of Vista was so low - I've seen estimates of 20%. Yet you can only do an in situ upgrade from Vista. So if you're one of the 80% who never moved from XP, you're out of luck. Or rather, Microsoft is, because that's who loses money if people don't upgrade. XP is a perfectly fine system, I would not have upgraded but for my "software wear-out" problem. Most not-very-technical users will be completely put off by the complication of a from-scratch installation, followed by reinstalling all of their apps, and in all probability just won't bother. So what was Microsoft thinking when they made this decision?
So, I booted from the memory stick, Windows 7 started and everything seemed fine. Then it put up a screen asking me which disk I wanted to use. When I selected my new drive, I got an error, "cannot create system partition". That's dumb, I thought, surely a from-scratch installer should be able to partition and format a disk? Still, I was able to rig things up so that I booted from the old system, with the new disk connected. That way I could format the new disk from XP. Then I restarted the Windows 7 system again. The same thing happened.
It took a lot of frustration, much bad language, and a lot of googling before I found the solution to this. It turns out that since the early days of Vista, the installer has not been able to cope with having more than one drive or partition visible to it. If it does, it gives this misleading error message which gives absolutely no clue as to the problem, and which will certainly stop most people dead in their tracks. Yet MS had at least a year to fix this, preferably so installer could handle multiple connected drives, but failing that at least to give a better error message. Once again, what were they thinking?
Once I had this figured out and unplugged everything except the target drive, everything went fine. The system installed remarkably quickly, and subsequent reinstallation of all my apps went smoothly too. It took quite a while for further problems to emerge.
Meanwhile, I hadn't given up on the disk mounting problem. I picked up various tools - stubby screwdriver, offset screwdriver - from my local TrueValue, but none of them could get all of the hidden screws. Finally a trip to Fry's got me a tiny ratchet offset screwdriver with interchangeable bits. Using that, and a lot of missing flesh from my hands, I was able to remove the drive and install the new one. Still I wondered how on earth they did this at the factory? Then by chance I tidied my office and found the original hardware manual for the machine. This revealed the secret latch which allows the whole drive bay to be dismantled and removed. Of course it is much easier that way, but there would be no way to identify the secret latch without the manual.
Windows 7 ran smoothly for a while, but then I ran into a very odd problem. Double clicking on certain system-ish folders gives the message "Access Denied". Now, I am the only user of the machine, and of course have Administrator privilege, so what is going on? I tried changing access rights, but of course since I didn't have access in the first place, that didn't work. Finally, some extensive googling found a way round this, which involves going into the deepest depths of "advanced options" in the security screen, and changing ownership of the parent directory. It's at times like this that Linux is so attractive - although the commands are arcane, it is always reasonably easy to figure out what is going on. With Windows, it's either easy or next-to-impossible. The dumbed-down user interfaces give you few alternatives when things are partly broken.
So now, my new system seems to be working fine, and does very nearly everything my old one did. (There are still a few things to reinstall). One mystery is a system-created "shortcut" (i.e. softlink) which points to itself. This confuses Explorer no end, but since I have no idea what the folder is for, I daren't delete it. I also still haven't managed to find the cache directory for IE8 - I've found one of them, but only some files show up there. That's another mystery.
Subscribe to:
Posts (Atom)