Fails in jruby (--1.9 ) because of incompatible encoding

[atechie]

Exact error was -

Encoding::CompatibilityError: incompatible encoding regexp match (UTF-8 regexp with ASCII-8BIT string)
from org/jruby/RubyString.java:2853:in gsub' from /home/prashant/.rvm/gems/jruby-1.6.4/gems/mojibake-1.0.0/lib/mojibake/mapper.rb:127:inrecover'

This was when I tried to recover from the content of a file, after reading the content its content was set to ASCII-8BIT , I did not force_encode to iso or windows-1252 because characters were mixed.

[dekellum]

At minimum we should improve the error handling/messaging. The characters of the mojibake can not be represented in ASCII (7 bit). The regex is built as UTF-8, so can you try calling .encode( "UTF-8" ) on the input before calling recover? This is what the bin/mojibake cli does. Does that not work for you input file?

[atechie]

Thanks dekellum . One clarification though - since input string is ASCII-8BIT , I think I should do .force_encoding("UTF-8") and not .encode("UTF-8").
ref- http://redmine.ruby-lang.org/issues/2313

[dekellum]

OK, so I should probably add an --input-encoding option or the like for the command line tool and do something like:

IO.read( input ).encode( input_encoding, 'UTF-8' )

Note that if input_encoding == 'UTF-8' than this is equivalent to .force_encoding( 'UTF-8' )

If your input is encoded as Windows-1252 (or ISO-8859-1, a practical subset) then you should:

IO.read( input ).encode( 'Windows-1252', 'UTF-8' )

In any case I think the regex is appropriately defined in UTF-8, thus the recover method should be documented to take a UTF-8 string, and/or I should add input.encode( "UTF-8" ) to recover itself.

Could you let me know if you are able to recover your text satisfactorily with any selected input encoding?