Saturday 26 November 2016

How to create an A0 VM instance on Azure

So I played with Azure hosting today.
You can currently get a free trial and e.g. £125 of credit.

But I thought I would try a really cheap server £9->£10/month server to play around with - i.e. the "A0" instance, to see what was possible on a low spec vm.

But, when you create a new VM, the A0 build doesn't seem to be an option.

So I selected the cheapest option they gave me and tried to resize it.
But that gave an error saying something about it not being available due to premium storage.

So, what I had to do was delete that VM and all the associated resources. Then:

1. create a new VM, but make sure you choose "HDD" and not "SSD" for the non-premium storage.
2. choose the A1 size (not the A0 size at this point, that wouldn't come up).
3. wait for it to finish deploying and enter the running state
4. click on the VM and stop it, then wait for it to enter the deallocated state
5. click on "size" for the VN and resize it A0

Obvious huh?

An A0 instance is very low spec, so I wouldn't recommend running windows on it due to the UI-heavy interface of windows for administration. Better to run linux, or perhaps try the windows nano server.
Also, be careful of additional costs like storage, you may want to resize them down too, and with all these cloud services, there are plenty of other costs/limits for bandwidth etc. so take a look at your billing daily until you know what's what.
Currently, if you are on a free trial, then according to an MS post " you will not be charged for your free trial at any point unless you explicitly take the action of upgrading to a paid subscription. If you do nothing, the trial will just end without charging your card." 

Sunday 12 June 2016

Route planning, GPX file import, mobile/phone - Tyre and Osmand

this is quick, unformatted reminder for me, and hopefully helpful to bikers/car enthusiasts/cyclists and other route planning types.

i was struggling for a while to find a combination of a decent GPS route planning/mapping tool and a sat nav phone app that could read it.

the route planning tools were a pain, including google maps which only lets you set urm 8 waypoints atm?

the sat nav apps that supported gpx files, e.g. copilot insisted on changing the route on me and bypassing waypoints i had set.


anyway, long story short, i found that the best combination so far seems to be "Tyre" on the PC and "Osmand" on the mobile.

1. download Tyre, note - windows only and must have a later version of IE installed. 
http://www.tyretotravel.com/download-tyre/download-tyre-basic/
alternatively if you don't have a PC, try e.g. https://www.motogoloco.com
tyre is better than anything online atm though 2. 
a.in tyre, right click on a road, and choose "add new waypoint (immediately)" repeatedly until you have built your route.
b. save this in the standard format (better for re-editing in case it doesn't like the gpx).
c. now save this in gpx format.

3. install Osmand on your device, the following instructions relate to android, but it is available for iphone

from one of these:
https://play.google.com/store/apps/details?id=net.osmand.plus
https://itunes.apple.com/app/apple-store/id934850257
https://www.amazon.com/OsmAnd-Maps-Navigation/dp/B00D0SEGMC


4. connect your phone to your windows PC, enable file browsing on your phone (click the usb message popup on the phone)

5. 
a. using windows explorer, browse to your phone storage and create a folder e.g. /OSMAND or /sdcard/OSMAND
b. now create a folder called tracks
c. copy the gpx file you created from your computer into the tracks folder

6. in the Osmand app go to menu/settings/general settings and click on data storage folder - change this folder to the folder that you created at 5a. 
e.g. /OSMAND or /sdcard/OSMAND

(the main point in 5+6 is to ensure that windows will be have permission to see the Osmand data folder without messing about with permissions. you can also e.g. email gpx files to yourself and open gpx files directly on your phone and osmand will import them)

7. 
a. in Osmand app, download the base (or overview) world map, and also the regional map covering your route, e.g. europe/uk/england. 
b. once those 2 maps are completely downloaded, go to menu/configure map/gpx track...   then select the track that you want, and select current track
(if you load another track later, you may wish to deselect current track then).



so that's just a quick run down. mostly for me for when i lose my phone and don't remember how to set all this up )

hopefully i didn't miss anything. 
when i am planning a route on tyre, i also have google earth/streetview up on the other screen so i can check out the roads.

don't forget to backup your routes to e.g. dropbox 


Sunday 16 March 2014

Getting a Dell C1760nw Color Printer up and running on Linux Mint / Ubuntu

I'm not so great at Linux, so I'm trying to use it as my home PC to get more familiar.

The biggest problem with Linux still seems to be getting the manufacturers to create drivers.

On the Dell support pages, there are no linux drivers for the C1760nw.

I found a post somewhere (I forget) that mentioned that C1760nw is basically a rebadaged Xerox Phaser 6000B.

So, I tried downloading the Xerox DEB driver, but that didn't work for me. (Edit: a few people have said they had no problems with the DEB driver - see comments here, so try that first!) So I downloaded the RPM print driver here (filename 6000_6010_rpm_1.01_20110222.zip)

I then opened the zip, extracted the contained Xerox_Phaser_6000B.ppd.gz to my home directory - as i am the only user. Which gives you the pdd file.
Then I extracted the contents of the RPM and found they contained a usr folder.
As root user, I then copied/merged the usr folder contents to /usr.

I then went to the linux printer setup at http://localhost:631
and added the printer:
Other network printers - Internet Printing Protocol (ipp)
Connection socket://192.168.0.100  (where 192.168.0.100 is my printer IP address)
Name - Dell
Provide a PDD file - browsed to and selected the ppd file I extracted earlier
And clicked Add Printer

I then fired up the printer gui with system-config-printer, clicked properties and tried to print a test page.

At which point, I got a printer status like this:
File /usr/lib/cups/filter/... has insecure permissions

(which was obviously the files I had extracted).

A bit of googling revealed I had to set the permissions up on the cups file in question, but being the only user, I just did the directory:

sudo chmod 755 -R /usr/lib/cups/filter
sudo chown -R root /usr/lib/cups/filter

So, I tried another test page, things went a bit further, and then I got this status:
Idle - "Rendering completed"

More googling revealed that I was missing some libraries:

sudo apt-get install ia32-libs

Another attempt and the print test finally page popped out!

30 mins to install a printer :c( 

Let's hope my Linux experience gets better. I do like Mint though, gotta say.

These replacement cartridges look cheap: amazon

Thursday 29 November 2012

Advanced jqGrid Search Functionality with LINQ (and EntityFramework) and ASP.NET MVC.


Note: there is a new version of jqGrid built specifically for MVC, but it's a commercial version which I am unfamiliar with. This article relates to the standard, free version.

Disclaimer: this code is not yet in production, and so is yet to be thoroughly tested. There may also be a much simpler, entirely different approach - if so, I'd love to know.

Special thanks to the jqGrid expert Oleg @ ok-soft-gmbh.com for the fundamental groupOp/filter code.


Ok, so you have built your jqGrid, and all is well, but:

  1. You need to add search functionality.
  2. You need to have full search functionality because you are using the jqGrid options multipleSearch: true, and multipleGroup: true, so you will be and/or-ing multiple search clauses.
  3. You don't want to code a specific search code for every jqGrid in your site, you want some simple shared functionality for the search that can be re-used.
  4. You want to be able to handle any text field name and value to support (3).
  5. You are using MVC and LINQ.

The most fundamental problem here is that LINQ will only allow type-safe queries. Out-of-the-box, this means that you cannot dynamically construct queries on the fly.
This is a problem for jqGrid searches from multiple sources, as you do not know the types (or names) for the where clause until run-time, or even the parameter value.

For example, normally in linq, you would write:

...where customer.id = 5

But we need something more like:

...where "customer.id = 5"

The most direct solution to this problem would probably be with expression trees. However, expression trees are rather complex, hard to read, and would become more involved to handle every possible where clause scenario.

Fortunately, there is a solution to this, the system.linq.dynamic class. This isn't part of the standard framework, but you can download and compile it yourself under the Microsoft Public Licence. For more information, see ScottGu's blog:


Ok, assuming you have built that dll, and added a reference in your project and a using statement in your controller, how are we going to build our where clause from what jqGrid provides us?

Well first we'll create some jqGrid support classes to provide plumbing (I will refer to this file later as the jqGrid support classes file):

    public class JqSearchIn
    {
        public string sidx { get; set; }
        public string sord { get; set; }
        public int page { get; set; }
        public int rows { get; set; }
        public bool _search { get; set; }
        public string searchField { get; set; }
        public string searchOper { get; set; }
        public string searchString { get; set; }
        public string filters { get; set; }

       // ignore this for now... all will become clear
        public WhereClause GenerateWhereClause(Type targetSearchType)
        {
            return new WhereClauseGenerator().Generate(_search, filters, targetSearchType);
        }
    }

    public class JqGridSearchOut
    {
        public int total { get; set; }
        public int page { get; set; }
        public int records { get; set; }
        public Array rows { get; set; }
    }

    public class JqGridFilter
    {
        public GroupOp groupOp { get; set; }
        public List<JqGridRule> rules { get; set; }
        public List<JqGridFilter> groups { get; set; }
    }

    public class JqGridRule
    {
        public string field { get; set; }
        public Operations op { get; set; }
        public string data { get; set; }
    }

    public enum GroupOp
    {
        AND,
        OR
    }

    public enum Operations
    {
        eq, // "equal"
        ne, // "not equal"
        lt, // "less"
        le, // "less or equal"
        gt, // "greater"
        ge, // "greater or equal"
        bw, // "begins with"
        bn, // "does not begin with"
        ew, // "ends with"
        en, // "does not end with"
        cn, // "contains"
        nc  // "does not contain"
        //in, // "in"
        //ni // "not in"
    }


Having done that, we can now create the signature for our controller, so assuming your jqGrid has an Ajax call in it's setup, like this:

    grid.jqGrid({
        height: 300,
        url: ajxGetJqGridCustomers,
        ...


Then the controller signature would look like this:

public JsonResult GetJqGridCustomers (JqSearchIn si)

(Ideally this should have a [HttpPost] above it, but I'll leave that up to you once you have everything else working).

Note that in MVC, the JqSearchIn parameter will auto-magically map correctly to populate the class.

Ok, having got that far, we can now start thinking about how to generate an appropriate where clause for all that search information the JqSearchIn has been populated with. Fundamentally, we want to do something like this (note the type being passed in, which is explained later):

var wc = si.GenerateWhereClause(typeof(Customer));

// I'm using entity framework here, but it could just as well
// be straight linq (i.e. forget the .db part)
customers = db.Customers.Where(wc);

But we're going to have to use parameters (remember our code must still be converted so it is (a) type-safe, and (b) parameterised against injection), so we're going to rewrite that like this:

customers = db.Customers.Where(wc.Clause, wc.FormatObjects);


We will add the object class to the jqGrid support classes file:

    public class WhereClause
    {
        public string Clause { get; set; }
        public object[] FormatObjects { get; set; }
    }


We can now create the clause generator and other associated methods (again I would suggest using the same jqGrid support classes file):


    // if you make this static, then consider concurrency issues
    public class WhereClauseGenerator
    {

        private List<object> _formatObjects;

        public WhereClause Generate(bool _search, string filters, Type targetSearchType )
        {
            _formatObjects = new List<object>();
           


            return new WhereClause()
                         {
                             Clause = _search && !String.IsNullOrEmpty(filters) ?
                                        ParseFilter(new JavaScriptSerializer()
                                             .Deserialize<JqGridFilter>(filters),targetSearchType).ToString()
                                        : String.Empty,
                             FormatObjects = _formatObjects.ToArray()
                         };
        }



        private readonly string[] FormatMapping = {
            // 0 = field name,
            // 1 = total no of formats/params so far - 1
            // F = replaced by fmAdd
            "({0} = @{1}{F})",               // "eq" - equal
            "({0} <> @{1}{F})",              // "ne" - not equal
            "({0} < @{1}{F})",               // "lt" - less than
            "({0} <= @{1}{F})",              // "le" - less than or equal to
            "({0} > @{1}{F})",               // "gt" - greater than
            "({0} >= @{1}{F})",              // "ge" - greater than or equal to
            "({0}.StartsWith(@{1}){F})",     // "bw" - begins with
            "(!{0}.StartsWith(@{1}){F})",    // "bn" - does not begin with
            "({0}.EndsWith(@{1}){F})",       // "ew" - ends with
            "(!{0}.EndsWith(@{1}){F})",      // "en" - does not end with
            "({0}.Contains(@{1}){F})",       // "cn" - contains
            "(!{0}.Contains(@{1}){F})"       // "nc" - does not contain
        };


        private readonly string[] NullValueFormatMapping = {
            // 0 = field name
            "({0} = NULL)",     // "eq" - equal
            "({0} != NULL)",    // "ne" - not equal
            "(1=0)",            // "lt" - less than
            "(1=1)",            // "le" - less than or equal to
            "({0} != NULL)",    // "gt" - greater than
            "(1=1)",            // "ge" - greater than or equal to
            "({0} != NULL)",    // "bw" - begins with
            "({0} != NULL)",    // "bn" - does not begin with
            "({0} != NULL)",    // "ew" - ends with
            "({0} != NULL)",    // "en" - does not end with
            "({0} != NULL)",    // "cn" - contains
            "({0} != NULL)"     // "nc" - does not contain
        };



        private StringBuilder ParseRule(ICollection<JqGridRule> rules, GroupOp groupOp, Type targetSearchType)
        {
            if (rules == null || rules.Count == 0)
                return null;

            var sb = new StringBuilder();
            bool firstRule = true;
            var props = targetSearchType.GetProperties().ToDictionary(p => p.Name, p => p.PropertyType);

            foreach (var rule in rules)
            {
                if (!firstRule)
                    // skip groupOp before the first rule
                    sb.Append(groupOp);
                else
                    firstRule = false;

                // get the object type of the rule
                Type ruleParseType;
                bool emptyNullable = false;
                try
                {
                    Type ruleType = ruleParseType = props[rule.field];

                    if (ruleType.IsGenericType && ruleType.GetGenericTypeDefinition() == typeof(Nullable<>))
                    {
                        if (rule.data == "")
                            emptyNullable = true;
                        ruleParseType = Nullable.GetUnderlyingType(ruleType);
                    }

                }
                catch(KeyNotFoundException) {
                    throw new ArgumentOutOfRangeException(rule.field + " is not a property of type "
                                                    + targetSearchType);
                }

                   
                // parse it in as the correct object type
                var fmAdd = "";
                if (ruleParseType == typeof(string))
                {
                    _formatObjects.Add(rule.data);
                    if (rule.data == "")
                        fmAdd = " OR {0} = NULL";
                }
                else
                {
                    if (emptyNullable)
                        _formatObjects.Add(null);
                    else
                    {
                        var parseMethod = ruleParseType.GetMethod("Parse", new[] {typeof (string)});
                        if (parseMethod != null)
                            _formatObjects.Add(parseMethod.Invoke(props[rule.field], new object[] {rule.data}));
                        else
                            throw new ArgumentOutOfRangeException(rule.field +
                                                 " is not a string and cannot be parsed either!!");
                    }
                }

                string fm = emptyNullable ? NullValueFormatMapping[(int) rule.op]
                                            : FormatMapping[(int) rule.op].Replace("{F}",fmAdd);

                sb.AppendFormat(fm, rule.field, _formatObjects.Count-1);

            }
            return sb.Length > 0 ? sb : null;
        }

        private void AppendWithBrackets(StringBuilder dest, StringBuilder src)
        {
            if (src == null || src.Length == 0)
                return;

            if (src.Length > 2 && src[0] != '(' && src[src.Length - 1] != ')')
            {
                dest.Append('(');
                dest.Append(src);
                dest.Append(')');
            }
            else
            {
                // verify that no other '(' and ')' exist in the b. so that
                // we have no case like src = "(x < 0) OR (y > 0)"
                for (int i = 1; i < src.Length - 1; i++)
                {
                    if (src[i] != '(' && src[i] != ')') continue;
                    dest.Append('(');
                    dest.Append(src);
                    dest.Append(')');
                    return;
                }
                dest.Append(src);
            }
        }

        private StringBuilder ParseFilter(ICollection<JqGridFilter> groups, GroupOp groupOp, Type targetSearchType)
        {
            if (groups == null || groups.Count == 0)
                return null;

            var sb = new StringBuilder();
            bool firstGroup = true;
            foreach (var group in groups)
            {
                var sbGroup = ParseFilter(group, targetSearchType);
                if (sbGroup == null || sbGroup.Length == 0)
                    continue;

                if (!firstGroup)
                    // skip groupOp before the first group
                    sb.Append(groupOp);
                else
                    firstGroup = false;

                sb.EnsureCapacity(sb.Length + sbGroup.Length + 2);
                AppendWithBrackets(sb, sbGroup);
            }
            return sb;
        }


        private StringBuilder ParseFilter(JqGridFilter filters, Type targetSearchType)
        {

            var parsedRules = ParseRule(filters.rules, filters.groupOp, targetSearchType);
            var parsedGroups = ParseFilter(filters.groups, filters.groupOp, targetSearchType);

            if (parsedRules != null && parsedRules.Length > 0)
            {
                if (parsedGroups != null && parsedGroups.Length > 0)
                {
                    var groupOpStr = filters.groupOp.ToString();
                    var sb = new StringBuilder(parsedRules.Length + parsedGroups.Length + groupOpStr.Length + 4);
                    AppendWithBrackets(sb, parsedRules);
                    sb.Append(groupOpStr);
                    AppendWithBrackets(sb, parsedGroups);
                    return sb;
                }
                return parsedRules;
            }
            return parsedGroups;
        }

    }


You'll notice several things in the above listing:

1. Most obviously, as expected, there is some recursion to build the clause (remember we are supporting multiple group and/ors). Note - for security you should never allow jgGrid's built-in showquery option string to be passed into your query.
2. There are various format objects for the parameters to ensure the where clause parameter list is built appropriately.
3. Most importantly, a targetSearchType is passed in. This is the most interesting part.

As you recall, the initial call from the controller is as follows:

si.GenerateWhereClause(typeof(Customer));

Which further down the stack becomes gets passed in to ParseRule:

private StringBuilder ParseRule(ICollection<JqGridRule> rules, GroupOp groupOp, Type targetSearchType)

ParseRule then uses Reflection to get the underlying type of each property being searched against in the entity/class, in this case Customer. It also uses reflection to get the correct parse method for the type. This means that we can parse in the correct type and value to build the where clause. Or in other words, we can keep dynamic linq happy.


One last bit of plumbing is to add a view model for the output:

    public class CustomerItemsJqGridRowOut
    {
        public int Id { get; set; }
        public string FirstName { get; set; }
        public string LastName { get; set; }
    }


Ok, having done the tricky parts, all that remains is to head back to the controller and finish off by adding paging and the rest of the code:



        public JsonResult GetJqGridCustomers(JqSearchIn si)
        {
            int totalRecords;
            int startRow = (si.page * si.rows) + 1;
            int skip = (si.page > 0 ? si.page - 1 : 0) * si.rows;


            // note - these queries require "using System.Dynamic.Linq" library
            IQueryable<Customer> customers;
            if (si._search && !String.IsNullOrEmpty(si.filters))
            {
                var wc = si.GenerateWhereClause(typeof(Customer));

                customers = db.Customers.Where(wc.Clause, wc.FormatObjects);

                totalRecords = customers.Count();

                customers = customers
                    .OrderBy(si.sidx + " " + si.sord)
                    .Skip(skip)
                    .Take(si.rows);

            }
            else
            {
                customers = db.Customers;
               
                totalRecords = customers.Count();

                customers = customers
                    .OrderBy(si.sidx + " " + si.sord)
                    .Skip(skip)
                    .Take(si.rows);
            }

           
            var customersList = customers.ToList();
            var totalPages = (int)Math.Ceiling((float)totalRecords / si.rows);
           
            var grid = new JqGridSearchOut
            {
                total = totalPages,
                page = si.page,
                records = totalRecords,
                rows = (from cst in customersList
                        select new CustomerItemsJqGridRowOut()
                        {
                            Id = cst.Id,
                            FirstName = cst.FirstName,
                            LastName = cst.LastName
                        }).ToArray()
            };

            return Json(grid);
        }


Hopefully this should be of use to someone out there.

Edit - I should probably do something about the OrderBy() clause as that part is obviously not parameterised, and could be injection attacked.