Brain  Crist

Brain Crist

1603659600

Dart sound null safety: technical preview 2

Dart serves a special role in Flutter, powering developer features such as hot reload, and enabling multi-platform apps for mobile, desktop, and web via Dart’s flexible compiler technology. We strive to make the Dart language the most productive for Flutter app developers; for example, we added UI-as-code language constructs to optimize the Dart syntax for coding Flutter widget trees.

In June we offered a first tech preview of null safety for Dart. Today is another major milestone, which we’ve looked forward to for a while: We’re announcing a second tech preview of sound null safety, including support for the Flutter framework.

Null safety is a major new productivity feature that helps you avoid null exceptions, a class of bugs that are often hard to spot. As an added bonus, this feature also enables a range of performance improvements. We’re really looking forward to your feedback.

Why null safety?

Dart is a type-safe language. This means that when you get a variable of some type, the compiler can guarantee that it is of that type. But type safety by itself doesn’t guarantee that the variable is not null.

Null errors are very common. A search on GitHub leads to thousands of issues caused by unexpected nulls in Dart code, and even more thousands of commits trying to fix those issues. Try to see if you can spot the nullability problems in the following Flutter app, imagining that Config and WeatherService are backend services used by the app:

This app will certainly fail if getAppName() returns a null; in that case we’ll pass a null to the Text widget used in the title of AppBar.

But there are more subtle cases to consider: getTemperatures() could also return null. In that case the for-loop will fail. Or getTemperatures() could return a list as expected, but that list might contain null values, in which case we’ll call round() on null, and the app will fail.

The null safety feature makes these problems go away by validating your code as you’re typing:

Screenshot of the preceding code with null errors.

With null safety, Dart finds potential null errors in your code.

With null safety, you can reason about your code with more confidence. No more pesky runtime null dereferencing errors in deployed apps. Instead, you get static errors as you code.

#programming #dart #null-safety #announcements #flutter

What is GEEK

Buddha Community

Dart sound null safety: technical preview 2
최 호민

최 호민

1642390128

파이썬 코딩 무료 강의 - 이미지 처리, 얼굴 인식을 통한 캐릭터 씌우기를 해보아요

파이썬 코딩 무료 강의 (활용편6) - 이미지 처리, 얼굴 인식을 통한 캐릭터 씌우기를 해보아요

파이썬 무료 강의 (활용편6 - 이미지 처리)입니다.
OpenCV 를 이용한 다양한 이미지 처리 기법과 재미있는 프로젝트를 진행합니다.
누구나 볼 수 있도록 쉽고 재미있게 제작하였습니다. ^^

[소개]
(0:00:00) 0.Intro
(0:00:31) 1.소개
(0:02:18) 2.활용편 6 이미지 처리 소개

[OpenCV 전반전]
(0:04:36) 3.환경설정
(0:08:41) 4.이미지 출력
(0:21:51) 5.동영상 출력 #1 파일
(0:29:58) 6.동영상 출력 #2 카메라
(0:34:23) 7.도형 그리기 #1 빈 스케치북
(0:39:49) 8.도형 그리기 #2 영역 색칠
(0:42:26) 9.도형 그리기 #3 직선
(0:51:23) 10.도형 그리기 #4 원
(0:55:09) 11.도형 그리기 #5 사각형
(0:58:32) 12.도형 그리기 #6 다각형
(1:09:23) 13.텍스트 #1 기본
(1:17:45) 14.텍스트 #2 한글 우회
(1:24:14) 15.파일 저장 #1 이미지
(1:29:27) 16.파일 저장 #2 동영상
(1:39:29) 17.크기 조정
(1:50:16) 18.이미지 자르기
(1:57:03) 19.이미지 대칭
(2:01:46) 20.이미지 회전
(2:06:07) 21.이미지 변형 - 흑백
(2:11:25) 22.이미지 변형 - 흐림
(2:18:03) 23.이미지 변형 - 원근 #1
(2:27:45) 24.이미지 변형 - 원근 #2

[반자동 문서 스캐너 프로젝트]
(2:32:50) 25.미니 프로젝트 1 - #1 마우스 이벤트 등록
(2:42:06) 26.미니 프로젝트 1 - #2 기본 코드 완성
(2:49:54) 27.미니 프로젝트 1 - #3 지점 선 긋기
(2:55:24) 28.미니 프로젝트 1 - #4 실시간 선 긋기

[OpenCV 후반전]
(3:01:52) 29.이미지 변형 - 이진화 #1 Trackbar
(3:14:37) 30.이미지 변형 - 이진화 #2 임계값
(3:20:26) 31.이미지 변형 - 이진화 #3 Adaptive Threshold
(3:28:34) 32.이미지 변형 - 이진화 #4 오츠 알고리즘
(3:32:22) 33.이미지 변환 - 팽창
(3:41:10) 34.이미지 변환 - 침식
(3:45:56) 35.이미지 변환 - 열림 & 닫힘
(3:54:10) 36.이미지 검출 - 경계선
(4:05:08) 37.이미지 검출 - 윤곽선 #1 기본
(4:15:26) 38.이미지 검출 - 윤곽선 #2 찾기 모드
(4:20:46) 39.이미지 검출 - 윤곽선 #3 면적

[카드 검출 & 분류기 프로젝트]
(4:27:42) 40.미니프로젝트 2

[퀴즈]
(4:31:57) 41.퀴즈

[얼굴인식 프로젝트]
(4:41:25) 42.환경설정 및 기본 코드 정리
(4:54:48) 43.눈과 코 인식하여 도형 그리기
(5:10:42) 44.그림판 이미지 씌우기
(5:20:52) 45.캐릭터 이미지 씌우기
(5:33:10) 46.보충설명
(5:40:53) 47.마치며 (학습 참고 자료)
(5:42:18) 48.Outro


[학습자료]
수업에 필요한 이미지, 동영상 자료 링크입니다.

고양이 이미지 : https://pixabay.com/images/id-2083492/ 
크기 : 640 x 390  
파일명 : img.jpg

고양이 동영상 : https://www.pexels.com/video/7515833/ 
크기 : SD (360 x 640)  
파일명 : video.mp4

신문 이미지 : https://pixabay.com/images/id-350376/ 
크기 : 1280 x 853  
파일명 : newspaper.jpg

카드 이미지 1 : https://pixabay.com/images/id-682332/ 
크기 : 1280 x 1019  
파일명 : poker.jpg

책 이미지 : https://www.pexels.com/photo/1029807/ 
크기 : Small (640 x 853)  
파일명 : book.jpg

눈사람 이미지 : https://pixabay.com/images/id-1300089/ 
크기 : 1280 x 904  
파일명 : snowman.png

카드 이미지 2 : https://pixabay.com/images/id-161404/ 
크기 : 640 x 408  
파일명 : card.png

퀴즈용 동영상 : https://www.pexels.com/video/3121459/ 
크기 : HD (1280 x 720)  
파일명 : city.mp4

프로젝트용 동영상 : https://www.pexels.com/video/3256542/ 
크기 : Full HD (1920 x 1080)  
파일명 : face_video.mp4

프로젝트용 캐릭터 이미지 : https://www.freepik.com/free-vector/cute-animal-masks-video-chat-application-effect-filters-set_6380101.htm  
파일명 : right_eye.png (100 x 100), left_eye.png (100 x 100), nose.png (300 x 100)

무료 이미지 편집 도구 : https://pixlr.com/kr/
(Pixlr E -Advanced Editor)

#python #opencv 

Ruth  Nabimanya

Ruth Nabimanya

1643018220

SheetJS: Parser and Writer for Various Spreadsheet Formats

SheetJS

Parser and writer for various spreadsheet formats. Pure-JS cleanroom implementation from official specifications, related documents, and test files. Emphasis on parsing and writing robustness, cross-format feature compatibility with a unified JS representation, and ES3/ES5 browser compatibility back to IE6.

This is the community version. We also offer a pro version with performance enhancements, additional features like styling, and dedicated support.

Community Translations of this README:

Build Status

Supported File Formats

circo graph of format support

Diagram Legend (click to show)

graph legend

Installation

In the browser, just add a script tag:

<script lang="javascript" src="dist/xlsx.full.min.js"></script>

CDN Availability (click to show)

CDNURL
unpkghttps://unpkg.com/xlsx/
jsDelivrhttps://jsdelivr.com/package/npm/xlsx
CDNjshttps://cdnjs.com/libraries/xlsx
packdhttps://bundle.run/xlsx@latest?name=XLSX

unpkg makes the latest version available at:

<script src="https://unpkg.com/xlsx/dist/xlsx.full.min.js"></script>

With npm:

$ npm install xlsx

With bower:

$ bower install js-xlsx

JS Ecosystem Demos

The demos directory includes sample projects for:

Frameworks and APIs

Bundlers and Tooling

Platforms and Integrations

Other examples are included in the showcase.

Optional Modules

Optional features (click to show)

The node version automatically requires modules for additional features. Some of these modules are rather large in size and are only needed in special circumstances, so they do not ship with the core. For browser use, they must be included directly:

<!-- international support from js-codepage -->
<script src="dist/cpexcel.js"></script>

An appropriate version for each dependency is included in the dist/ directory.

The complete single-file version is generated at dist/xlsx.full.min.js

A slimmer build is generated at dist/xlsx.mini.min.js. Compared to full build:

  • codepage library skipped (no support for XLS encodings)
  • XLSX compression option not currently available
  • no support for XLSB / XLS / Lotus 1-2-3 / SpreadsheetML 2003
  • node stream utils removed

Webpack and Browserify builds include optional modules by default. Webpack can be configured to remove support with resolve.alias:

  /* uncomment the lines below to remove support */
  resolve: {
    alias: { "./dist/cpexcel.js": "" } // <-- omit international support
  }

ECMAScript 5 Compatibility

Since the library uses functions like Array#forEach, older browsers require shims to provide missing functions.

To use the shim, add the shim before the script tag that loads xlsx.js:

<!-- add the shim first -->
<script type="text/javascript" src="shim.min.js"></script>
<!-- after the shim is referenced, add the library -->
<script type="text/javascript" src="xlsx.full.min.js"></script>

The script also includes IE_LoadFile and IE_SaveFile for loading and saving files in Internet Explorer versions 6-9. The xlsx.extendscript.js script bundles the shim in a format suitable for Photoshop and other Adobe products.

Philosophy

Philosophy (click to show)

Prior to SheetJS, APIs for processing spreadsheet files were format-specific. Third-party libraries either supported one format, or they involved a separate set of classes for each supported file type. Even though XLSB was introduced in Excel 2007, nothing outside of SheetJS or Excel supported the format.

To promote a format-agnostic view, SheetJS starts from a pure-JS representation that we call the "Common Spreadsheet Format". Emphasizing a uniform object representation enables new features like format conversion (reading an XLSX template and saving as XLS) and circumvents the mess of classes. By abstracting the complexities of the various formats, tools need not worry about the specific file type!

A simple object representation combined with careful coding practices enables use cases in older browsers and in alternative environments like ExtendScript and Web Workers. It is always tempting to use the latest and greatest features, but they tend to require the latest versions of browsers, limiting usability.

Utility functions capture common use cases like generating JS objects or HTML. Most simple operations should only require a few lines of code. More complex operations generally should be straightforward to implement.

Excel pushes the XLSX format as default starting in Excel 2007. However, there are other formats with more appealing properties. For example, the XLSB format is spiritually similar to XLSX but files often tend up taking less than half the space and open much faster! Even though an XLSX writer is available, other format writers are available so users can take advantage of the unique characteristics of each format.

The primary focus of the Community Edition is correct data interchange, focused on extracting data from any compatible data representation and exporting data in various formats suitable for any third party interface.

Parsing Workbooks

For parsing, the first step is to read the file. This involves acquiring the data and feeding it into the library. Here are a few common scenarios:

nodejs read a file (click to show)

readFile is only available in server environments. Browsers have no API for reading arbitrary files given a path, so another strategy must be used.

if(typeof require !== 'undefined') XLSX = require('xlsx');
var workbook = XLSX.readFile('test.xlsx');
/* DO SOMETHING WITH workbook HERE */

Photoshop ExtendScript read a file (click to show)

readFile wraps the File logic in Photoshop and other ExtendScript targets. The specified path should be an absolute path:

#include "xlsx.extendscript.js"
/* Read test.xlsx from the Documents folder */
var workbook = XLSX.readFile(Folder.myDocuments + '/' + 'test.xlsx');
/* DO SOMETHING WITH workbook HERE */

The extendscript demo includes a more complex example.

Browser read TABLE element from page (click to show)

The table_to_book and table_to_sheet utility functions take a DOM TABLE element and iterate through the child nodes.

var workbook = XLSX.utils.table_to_book(document.getElementById('tableau'));
/* DO SOMETHING WITH workbook HERE */

Multiple tables on a web page can be converted to individual worksheets:

/* create new workbook */
var workbook = XLSX.utils.book_new();

/* convert table 'table1' to worksheet named "Sheet1" */
var ws1 = XLSX.utils.table_to_sheet(document.getElementById('table1'));
XLSX.utils.book_append_sheet(workbook, ws1, "Sheet1");

/* convert table 'table2' to worksheet named "Sheet2" */
var ws2 = XLSX.utils.table_to_sheet(document.getElementById('table2'));
XLSX.utils.book_append_sheet(workbook, ws2, "Sheet2");

/* workbook now has 2 worksheets */

Alternatively, the HTML code can be extracted and parsed:

var htmlstr = document.getElementById('tableau').outerHTML;
var workbook = XLSX.read(htmlstr, {type:'string'});

Browser download file (ajax) (click to show)

Note: for a more complete example that works in older browsers, check the demo at http://oss.sheetjs.com/sheetjs/ajax.html. The xhr demo includes more examples with XMLHttpRequest and fetch.

var url = "http://oss.sheetjs.com/test_files/formula_stress_test.xlsx";

/* set up async GET request */
var req = new XMLHttpRequest();
req.open("GET", url, true);
req.responseType = "arraybuffer";

req.onload = function(e) {
  var workbook = XLSX.read(req.response);

  /* DO SOMETHING WITH workbook HERE */
}

req.send();

Browser drag-and-drop (click to show)

For modern browsers, Blob#arrayBuffer can read data from files:

async function handleDropAsync(e) {
  e.stopPropagation(); e.preventDefault();
  const f = evt.dataTransfer.files[0];
  const data = await f.arrayBuffer();
  const workbook = XLSX.read(data);

  /* DO SOMETHING WITH workbook HERE */
}
drop_dom_element.addEventListener('drop', handleDropAsync, false);

For maximal compatibility, the FileReader API should be used:

function handleDrop(e) {
  e.stopPropagation(); e.preventDefault();
  var f = e.dataTransfer.files[0];
  var reader = new FileReader();
  reader.onload = function(e) {
    var workbook = XLSX.read(e.target.result);

    /* DO SOMETHING WITH workbook HERE */
  };
  reader.readAsArrayBuffer(f);
}
drop_dom_element.addEventListener('drop', handleDrop, false);

Browser file upload form element (click to show)

Data from file input elements can be processed using the same APIs as in the drag-and-drop example.

Using Blob#arrayBuffer:

async function handleFileAsync(e) {
  const file = e.target.files[0];
  const data = await file.arrayBuffer();
  const workbook = XLSX.read(data);

  /* DO SOMETHING WITH workbook HERE */
}
input_dom_element.addEventListener('change', handleFileAsync, false);

Using FileReader:

function handleFile(e) {
  var files = e.target.files, f = files[0];
  var reader = new FileReader();
  reader.onload = function(e) {
    var workbook = XLSX.read(e.target.result);

    /* DO SOMETHING WITH workbook HERE */
  };
  reader.readAsArrayBuffer(f);
}
input_dom_element.addEventListener('change', handleFile, false);

The oldie demo shows an IE-compatible fallback scenario.

More specialized cases, including mobile app file processing, are covered in the included demos

Parsing Examples

Note that older versions of IE do not support HTML5 File API, so the Base64 mode is used for testing.

Get Base64 encoding on OSX / Windows (click to show)

On OSX you can get the Base64 encoding with:

$ <target_file base64 | pbcopy

On Windows XP and up you can get the Base64 encoding using certutil:

> certutil -encode target_file target_file.b64

(note: You have to open the file and remove the header and footer lines)

Streaming Read

Why is there no Streaming Read API? (click to show)

The most common and interesting formats (XLS, XLSX/M, XLSB, ODS) are ultimately ZIP or CFB containers of files. Neither format puts the directory structure at the beginning of the file: ZIP files place the Central Directory records at the end of the logical file, while CFB files can place the storage info anywhere in the file! As a result, to properly handle these formats, a streaming function would have to buffer the entire file before commencing. That belies the expectations of streaming, so we do not provide any streaming read API.

When dealing with Readable Streams, the easiest approach is to buffer the stream and process the whole thing at the end. This can be done with a temporary file or by explicitly concatenating the stream:

Explicitly concatenating streams (click to show)

var fs = require('fs');
var XLSX = require('xlsx');
function process_RS(stream/*:ReadStream*/, cb/*:(wb:Workbook)=>void*/)/*:void*/{
  var buffers = [];
  stream.on('data', function(data) { buffers.push(data); });
  stream.on('end', function() {
    var buffer = Buffer.concat(buffers);
    var workbook = XLSX.read(buffer, {type:"buffer"});

    /* DO SOMETHING WITH workbook IN THE CALLBACK */
    cb(workbook);
  });
}

More robust solutions are available using modules like concat-stream.

Writing to filesystem first (click to show)

This example uses tempfile to generate file names:

var fs = require('fs'), tempfile = require('tempfile');
var XLSX = require('xlsx');
function process_RS(stream/*:ReadStream*/, cb/*:(wb:Workbook)=>void*/)/*:void*/{
  var fname = tempfile('.sheetjs');
  console.log(fname);
  var ostream = fs.createWriteStream(fname);
  stream.pipe(ostream);
  ostream.on('finish', function() {
    var workbook = XLSX.readFile(fname);
    fs.unlinkSync(fname);

    /* DO SOMETHING WITH workbook IN THE CALLBACK */
    cb(workbook);
  });
}

Working with the Workbook

The full object format is described later in this README.

Reading a specific cell (click to show)

This example extracts the value stored in cell A1 from the first worksheet:

var first_sheet_name = workbook.SheetNames[0];
var address_of_cell = 'A1';

/* Get worksheet */
var worksheet = workbook.Sheets[first_sheet_name];

/* Find desired cell */
var desired_cell = worksheet[address_of_cell];

/* Get the value */
var desired_value = (desired_cell ? desired_cell.v : undefined);

Adding a new worksheet to a workbook (click to show)

This example uses XLSX.utils.aoa_to_sheet to make a sheet and XLSX.utils.book_append_sheet to append the sheet to the workbook:

var ws_name = "SheetJS";

/* make worksheet */
var ws_data = [
  [ "S", "h", "e", "e", "t", "J", "S" ],
  [  1 ,  2 ,  3 ,  4 ,  5 ]
];
var ws = XLSX.utils.aoa_to_sheet(ws_data);

/* Add the worksheet to the workbook */
XLSX.utils.book_append_sheet(wb, ws, ws_name);

Creating a new workbook from scratch (click to show)

The workbook object contains a SheetNames array of names and a Sheets object mapping sheet names to sheet objects. The XLSX.utils.book_new utility function creates a new workbook object:

/* create a new blank workbook */
var wb = XLSX.utils.book_new();

The new workbook is blank and contains no worksheets. The write functions will error if the workbook is empty.

Parsing and Writing Examples

https://sheetjs.com/demos/modify.html read + modify + write files

https://github.com/SheetJS/sheetjs/blob/HEAD/bin/xlsx.njs node

The node version installs a command line tool xlsx which can read spreadsheet files and output the contents in various formats. The source is available at xlsx.njs in the bin directory.

Some helper functions in XLSX.utils generate different views of the sheets:

  • XLSX.utils.sheet_to_csv generates CSV
  • XLSX.utils.sheet_to_txt generates UTF16 Formatted Text
  • XLSX.utils.sheet_to_html generates HTML
  • XLSX.utils.sheet_to_json generates an array of objects
  • XLSX.utils.sheet_to_formulae generates a list of formulae

Writing Workbooks

For writing, the first step is to generate output data. The helper functions write and writeFile will produce the data in various formats suitable for dissemination. The second step is to actual share the data with the end point. Assuming workbook is a workbook object:

nodejs write a file (click to show)

XLSX.writeFile uses fs.writeFileSync in server environments:

if(typeof require !== 'undefined') XLSX = require('xlsx');
/* output format determined by filename */
XLSX.writeFile(workbook, 'out.xlsb');
/* at this point, out.xlsb is a file that you can distribute */

Photoshop ExtendScript write a file (click to show)

writeFile wraps the File logic in Photoshop and other ExtendScript targets. The specified path should be an absolute path:

#include "xlsx.extendscript.js"
/* output format determined by filename */
XLSX.writeFile(workbook, 'out.xlsx');
/* at this point, out.xlsx is a file that you can distribute */

The extendscript demo includes a more complex example.

Browser add TABLE element to page (click to show)

The sheet_to_html utility function generates HTML code that can be added to any DOM element.

var worksheet = workbook.Sheets[workbook.SheetNames[0]];
var container = document.getElementById('tableau');
container.innerHTML = XLSX.utils.sheet_to_html(worksheet);

Browser upload file (ajax) (click to show)

A complete example using XHR is included in the XHR demo, along with examples for fetch and wrapper libraries. This example assumes the server can handle Base64-encoded files (see the demo for a basic nodejs server):

/* in this example, send a base64 string to the server */
var wopts = { bookType:'xlsx', bookSST:false, type:'base64' };

var wbout = XLSX.write(workbook,wopts);

var req = new XMLHttpRequest();
req.open("POST", "/upload", true);
var formdata = new FormData();
formdata.append('file', 'test.xlsx'); // <-- server expects `file` to hold name
formdata.append('data', wbout); // <-- `data` holds the base64-encoded data
req.send(formdata);

Browser save file (click to show)

XLSX.writeFile wraps a few techniques for triggering a file save:

  • URL browser API creates an object URL for the file, which the library uses by creating a link and forcing a click. It is supported in modern browsers.
  • msSaveBlob is an IE10+ API for triggering a file save.
  • IE_FileSave uses VBScript and ActiveX to write a file in IE6+ for Windows XP and Windows 7. The shim must be included in the containing HTML page.

There is no standard way to determine if the actual file has been downloaded.

/* output format determined by filename */
XLSX.writeFile(workbook, 'out.xlsb');
/* at this point, out.xlsb will have been downloaded */

Browser save file (compatibility) (click to show)

XLSX.writeFile techniques work for most modern browsers as well as older IE. For much older browsers, there are workarounds implemented by wrapper libraries.

FileSaver.js implements saveAs. Note: XLSX.writeFile will automatically call saveAs if available.

/* bookType can be any supported output type */
var wopts = { bookType:'xlsx', bookSST:false, type:'array' };

var wbout = XLSX.write(workbook,wopts);

/* the saveAs call downloads a file on the local machine */
saveAs(new Blob([wbout],{type:"application/octet-stream"}), "test.xlsx");

Downloadify uses a Flash SWF button to generate local files, suitable for environments where ActiveX is unavailable:

Downloadify.create(id,{
    /* other options are required! read the downloadify docs for more info */
    filename: "test.xlsx",
    data: function() { return XLSX.write(wb, {bookType:"xlsx", type:'base64'}); },
    append: false,
    dataType: 'base64'
});

The oldie demo shows an IE-compatible fallback scenario.

The included demos cover mobile apps and other special deployments.

Writing Examples

Streaming Write

The streaming write functions are available in the XLSX.stream object. They take the same arguments as the normal write functions but return a Readable Stream. They are only exposed in NodeJS.

  • XLSX.stream.to_csv is the streaming version of XLSX.utils.sheet_to_csv.
  • XLSX.stream.to_html is the streaming version of XLSX.utils.sheet_to_html.
  • XLSX.stream.to_json is the streaming version of XLSX.utils.sheet_to_json.

nodejs convert to CSV and write file (click to show)

var output_file_name = "out.csv";
var stream = XLSX.stream.to_csv(worksheet);
stream.pipe(fs.createWriteStream(output_file_name));

nodejs write JSON stream to screen (click to show)

/* to_json returns an object-mode stream */
var stream = XLSX.stream.to_json(worksheet, {raw:true});

/* the following stream converts JS objects to text via JSON.stringify */
var conv = new Transform({writableObjectMode:true});
conv._transform = function(obj, e, cb){ cb(null, JSON.stringify(obj) + "\n"); };

stream.pipe(conv); conv.pipe(process.stdout);

https://github.com/sheetjs/sheetaki pipes write streams to nodejs response.

Interface

XLSX is the exposed variable in the browser and the exported node variable

XLSX.version is the version of the library (added by the build script).

XLSX.SSF is an embedded version of the format library.

Parsing functions

XLSX.read(data, read_opts) attempts to parse data.

XLSX.readFile(filename, read_opts) attempts to read filename and parse.

Parse options are described in the Parsing Options section.

Writing functions

XLSX.write(wb, write_opts) attempts to write the workbook wb

XLSX.writeFile(wb, filename, write_opts) attempts to write wb to filename. In browser-based environments, it will attempt to force a client-side download.

XLSX.writeFileAsync(wb, filename, o, cb) attempts to write wb to filename. If o is omitted, the writer will use the third argument as the callback.

XLSX.stream contains a set of streaming write functions.

Write options are described in the Writing Options section.

Utilities

Utilities are available in the XLSX.utils object and are described in the Utility Functions section:

Importing:

  • aoa_to_sheet converts an array of arrays of JS data to a worksheet.
  • json_to_sheet converts an array of JS objects to a worksheet.
  • table_to_sheet converts a DOM TABLE element to a worksheet.
  • sheet_add_aoa adds an array of arrays of JS data to an existing worksheet.
  • sheet_add_json adds an array of JS objects to an existing worksheet.

Exporting:

  • sheet_to_json converts a worksheet object to an array of JSON objects.
  • sheet_to_csv generates delimiter-separated-values output.
  • sheet_to_txt generates UTF16 formatted text.
  • sheet_to_html generates HTML output.
  • sheet_to_formulae generates a list of the formulae (with value fallbacks).

Cell and cell address manipulation:

  • format_cell generates the text value for a cell (using number formats).
  • encode_row / decode_row converts between 0-indexed rows and 1-indexed rows.
  • encode_col / decode_col converts between 0-indexed columns and column names.
  • encode_cell / decode_cell converts cell addresses.
  • encode_range / decode_range converts cell ranges.

Common Spreadsheet Format

SheetJS conforms to the Common Spreadsheet Format (CSF):

General Structures

Cell address objects are stored as {c:C, r:R} where C and R are 0-indexed column and row numbers, respectively. For example, the cell address B5 is represented by the object {c:1, r:4}.

Cell range objects are stored as {s:S, e:E} where S is the first cell and E is the last cell in the range. The ranges are inclusive. For example, the range A3:B7 is represented by the object {s:{c:0, r:2}, e:{c:1, r:6}}. Utility functions perform a row-major order walk traversal of a sheet range:

for(var R = range.s.r; R <= range.e.r; ++R) {
  for(var C = range.s.c; C <= range.e.c; ++C) {
    var cell_address = {c:C, r:R};
    /* if an A1-style address is needed, encode the address */
    var cell_ref = XLSX.utils.encode_cell(cell_address);
  }
}

Cell Object

Cell objects are plain JS objects with keys and values following the convention:

KeyDescription
vraw value (see Data Types section for more info)
wformatted text (if applicable)
ttype: b Boolean, e Error, n Number, d Date, s Text, z Stub
fcell formula encoded as an A1-style string (if applicable)
Frange of enclosing array if formula is array formula (if applicable)
rrich text encoding (if applicable)
hHTML rendering of the rich text (if applicable)
ccomments associated with the cell
znumber format string associated with the cell (if requested)
lcell hyperlink object (.Target holds link, .Tooltip is tooltip)
sthe style/theme of the cell (if applicable)

Built-in export utilities (such as the CSV exporter) will use the w text if it is available. To change a value, be sure to delete cell.w (or set it to undefined) before attempting to export. The utilities will regenerate the w text from the number format (cell.z) and the raw value if possible.

The actual array formula is stored in the f field of the first cell in the array range. Other cells in the range will omit the f field.

Data Types

The raw value is stored in the v value property, interpreted based on the t type property. This separation allows for representation of numbers as well as numeric text. There are 6 valid cell types:

TypeDescription
bBoolean: value interpreted as JS boolean
eError: value is a numeric code and w property stores common name **
nNumber: value is a JS number **
dDate: value is a JS Date object or string to be parsed as Date **
sText: value interpreted as JS string and written as text **
zStub: blank stub cell that is ignored by data processing utilities **

Error values and interpretation (click to show)

ValueError Meaning
0x00#NULL!
0x07#DIV/0!
0x0F#VALUE!
0x17#REF!
0x1D#NAME?
0x24#NUM!
0x2A#N/A
0x2B#GETTING_DATA

Type n is the Number type. This includes all forms of data that Excel stores as numbers, such as dates/times and Boolean fields. Excel exclusively uses data that can be fit in an IEEE754 floating point number, just like JS Number, so the v field holds the raw number. The w field holds formatted text. Dates are stored as numbers by default and converted with XLSX.SSF.parse_date_code.

Type d is the Date type, generated only when the option cellDates is passed. Since JSON does not have a natural Date type, parsers are generally expected to store ISO 8601 Date strings like you would get from date.toISOString(). On the other hand, writers and exporters should be able to handle date strings and JS Date objects. Note that Excel disregards timezone modifiers and treats all dates in the local timezone. The library does not correct for this error.

Type s is the String type. Values are explicitly stored as text. Excel will interpret these cells as "number stored as text". Generated Excel files automatically suppress that class of error, but other formats may elicit errors.

Type z represents blank stub cells. They are generated in cases where cells have no assigned value but hold comments or other metadata. They are ignored by the core library data processing utility functions. By default these cells are not generated; the parser sheetStubs option must be set to true.

Dates

Excel Date Code details (click to show)

By default, Excel stores dates as numbers with a format code that specifies date processing. For example, the date 19-Feb-17 is stored as the number 42785 with a number format of d-mmm-yy. The SSF module understands number formats and performs the appropriate conversion.

XLSX also supports a special date type d where the data is an ISO 8601 date string. The formatter converts the date back to a number.

The default behavior for all parsers is to generate number cells. Setting cellDates to true will force the generators to store dates.

Time Zones and Dates (click to show)

Excel has no native concept of universal time. All times are specified in the local time zone. Excel limitations prevent specifying true absolute dates.

Following Excel, this library treats all dates as relative to local time zone.

Epochs: 1900 and 1904 (click to show)

Excel supports two epochs (January 1 1900 and January 1 1904). The workbook's epoch can be determined by examining the workbook's wb.Workbook.WBProps.date1904 property:

!!(((wb.Workbook||{}).WBProps||{}).date1904)

Sheet Objects

Each key that does not start with ! maps to a cell (using A-1 notation)

sheet[address] returns the cell object for the specified address.

Special sheet keys (accessible as sheet[key], each starting with !):

sheet['!ref']: A-1 based range representing the sheet range. Functions that work with sheets should use this parameter to determine the range. Cells that are assigned outside of the range are not processed. In particular, when writing a sheet by hand, cells outside of the range are not included

Functions that handle sheets should test for the presence of !ref field. If the !ref is omitted or is not a valid range, functions are free to treat the sheet as empty or attempt to guess the range. The standard utilities that ship with this library treat sheets as empty (for example, the CSV output is empty string).

When reading a worksheet with the sheetRows property set, the ref parameter will use the restricted range. The original range is set at ws['!fullref']

sheet['!margins']: Object representing the page margins. The default values follow Excel's "normal" preset. Excel also has a "wide" and a "narrow" preset but they are stored as raw measurements. The main properties are listed below:

Page margin details (click to show)

keydescription"normal""wide""narrow"
leftleft margin (inches)0.71.00.25
rightright margin (inches)0.71.00.25
toptop margin (inches)0.751.00.75
bottombottom margin (inches)0.751.00.75
headerheader margin (inches)0.30.50.3
footerfooter margin (inches)0.30.50.3
/* Set worksheet sheet to "normal" */
ws["!margins"]={left:0.7, right:0.7, top:0.75,bottom:0.75,header:0.3,footer:0.3}
/* Set worksheet sheet to "wide" */
ws["!margins"]={left:1.0, right:1.0, top:1.0, bottom:1.0, header:0.5,footer:0.5}
/* Set worksheet sheet to "narrow" */
ws["!margins"]={left:0.25,right:0.25,top:0.75,bottom:0.75,header:0.3,footer:0.3}

Worksheet Object

In addition to the base sheet keys, worksheets also add:

ws['!cols']: array of column properties objects. Column widths are actually stored in files in a normalized manner, measured in terms of the "Maximum Digit Width" (the largest width of the rendered digits 0-9, in pixels). When parsed, the column objects store the pixel width in the wpx field, character width in the wch field, and the maximum digit width in the MDW field.

ws['!rows']: array of row properties objects as explained later in the docs. Each row object encodes properties including row height and visibility.

ws['!merges']: array of range objects corresponding to the merged cells in the worksheet. Plain text formats do not support merge cells. CSV export will write all cells in the merge range if they exist, so be sure that only the first cell (upper-left) in the range is set.

ws['!outline']: configure how outlines should behave. Options default to the default settings in Excel 2019:

keyExcel featuredefault
aboveUncheck "Summary rows below detail"false
leftUncheck "Summary rows to the right of detail"false
  • ws['!protect']: object of write sheet protection properties. The password key specifies the password for formats that support password-protected sheets (XLSX/XLSB/XLS). The writer uses the XOR obfuscation method. The following keys control the sheet protection -- set to false to enable a feature when sheet is locked or set to true to disable a feature:

Worksheet Protection Details (click to show)

keyfeature (true=disabled / false=enabled)default
selectLockedCellsSelect locked cellsenabled
selectUnlockedCellsSelect unlocked cellsenabled
formatCellsFormat cellsdisabled
formatColumnsFormat columnsdisabled
formatRowsFormat rowsdisabled
insertColumnsInsert columnsdisabled
insertRowsInsert rowsdisabled
insertHyperlinksInsert hyperlinksdisabled
deleteColumnsDelete columnsdisabled
deleteRowsDelete rowsdisabled
sortSortdisabled
autoFilterFilterdisabled
pivotTablesUse PivotTable reportsdisabled
objectsEdit objectsenabled
scenariosEdit scenariosenabled
  • ws['!autofilter']: AutoFilter object following the schema:
type AutoFilter = {
  ref:string; // A-1 based range representing the AutoFilter table range
}

Chartsheet Object

Chartsheets are represented as standard sheets. They are distinguished with the !type property set to "chart".

The underlying data and !ref refer to the cached data in the chartsheet. The first row of the chartsheet is the underlying header.

Macrosheet Object

Macrosheets are represented as standard sheets. They are distinguished with the !type property set to "macro".

Dialogsheet Object

Dialogsheets are represented as standard sheets. They are distinguished with the !type property set to "dialog".

Workbook Object

workbook.SheetNames is an ordered list of the sheets in the workbook

wb.Sheets[sheetname] returns an object representing the worksheet.

wb.Props is an object storing the standard properties. wb.Custprops stores custom properties. Since the XLS standard properties deviate from the XLSX standard, XLS parsing stores core properties in both places.

wb.Workbook stores workbook-level attributes.

Workbook File Properties

The various file formats use different internal names for file properties. The workbook Props object normalizes the names:

File Properties (click to show)

JS NameExcel Description
TitleSummary tab "Title"
SubjectSummary tab "Subject"
AuthorSummary tab "Author"
ManagerSummary tab "Manager"
CompanySummary tab "Company"
CategorySummary tab "Category"
KeywordsSummary tab "Keywords"
CommentsSummary tab "Comments"
LastAuthorStatistics tab "Last saved by"
CreatedDateStatistics tab "Created"

For example, to set the workbook title property:

if(!wb.Props) wb.Props = {};
wb.Props.Title = "Insert Title Here";

Custom properties are added in the workbook Custprops object:

if(!wb.Custprops) wb.Custprops = {};
wb.Custprops["Custom Property"] = "Custom Value";

Writers will process the Props key of the options object:

/* force the Author to be "SheetJS" */
XLSX.write(wb, {Props:{Author:"SheetJS"}});

Workbook-Level Attributes

wb.Workbook stores workbook-level attributes.

Defined Names

wb.Workbook.Names is an array of defined name objects which have the keys:

Defined Name Properties (click to show)

KeyDescription
SheetName scope. Sheet Index (0 = first sheet) or null (Workbook)
NameCase-sensitive name. Standard rules apply **
RefA1-style Reference ("Sheet1!$A$1:$D$20")
CommentComment (only applicable for XLS/XLSX/XLSB)

Excel allows two sheet-scoped defined names to share the same name. However, a sheet-scoped name cannot collide with a workbook-scope name. Workbook writers may not enforce this constraint.

Workbook Views

wb.Workbook.Views is an array of workbook view objects which have the keys:

KeyDescription
RTLIf true, display right-to-left

Miscellaneous Workbook Properties

wb.Workbook.WBProps holds other workbook properties:

KeyDescription
CodeNameVBA Project Workbook Code Name
date1904epoch: 0/false for 1900 system, 1/true for 1904
filterPrivacyWarn or strip personally identifying info on save

Document Features

Even for basic features like date storage, the official Excel formats store the same content in different ways. The parsers are expected to convert from the underlying file format representation to the Common Spreadsheet Format. Writers are expected to convert from CSF back to the underlying file format.

Formulae

The A1-style formula string is stored in the f field. Even though different file formats store the formulae in different ways, the formats are translated. Even though some formats store formulae with a leading equal sign, CSF formulae do not start with =.

Representation of A1=1, A2=2, A3=A1+A2 (click to show)

{
  "!ref": "A1:A3",
  A1: { t:'n', v:1 },
  A2: { t:'n', v:2 },
  A3: { t:'n', v:3, f:'A1+A2' }
}

Shared formulae are decompressed and each cell has the formula corresponding to its cell. Writers generally do not attempt to generate shared formulae.

Cells with formula entries but no value will be serialized in a way that Excel and other spreadsheet tools will recognize. This library will not automatically compute formula results! For example, to compute BESSELJ in a worksheet:

Formula without known value (click to show)

{
  "!ref": "A1:A3",
  A1: { t:'n', v:3.14159 },
  A2: { t:'n', v:2 },
  A3: { t:'n', f:'BESSELJ(A1,A2)' }
}

Array Formulae

Array formulae are stored in the top-left cell of the array block. All cells of an array formula have a F field corresponding to the range. A single-cell formula can be distinguished from a plain formula by the presence of F field.

Array Formula examples (click to show)

For example, setting the cell C1 to the array formula {=SUM(A1:A3*B1:B3)}:

worksheet['C1'] = { t:'n', f: "SUM(A1:A3*B1:B3)", F:"C1:C1" };

For a multi-cell array formula, every cell has the same array range but only the first cell specifies the formula. Consider D1:D3=A1:A3*B1:B3:

worksheet['D1'] = { t:'n', F:"D1:D3", f:"A1:A3*B1:B3" };
worksheet['D2'] = { t:'n', F:"D1:D3" };
worksheet['D3'] = { t:'n', F:"D1:D3" };

Utilities and writers are expected to check for the presence of a F field and ignore any possible formula element f in cells other than the starting cell. They are not expected to perform validation of the formulae!

Formula Output Utility Function (click to show)

The sheet_to_formulae method generates one line per formula or array formula. Array formulae are rendered in the form range=formula while plain cells are rendered in the form cell=formula or value. Note that string literals are prefixed with an apostrophe ', consistent with Excel's formula bar display.

Formulae File Format Details (click to show)

Storage RepresentationFormatsReadWrite
A1-style stringsXLSX
RC-style stringsXLML and plain text
BIFF Parsed formulaeXLSB and all XLS formats 
OpenFormula formulaeODS/FODS/UOS
Lotus Parsed formulaeAll Lotus WK_ formats 

Since Excel prohibits named cells from colliding with names of A1 or RC style cell references, a (not-so-simple) regex conversion is possible. BIFF Parsed formulae and Lotus Parsed formulae have to be explicitly unwound. OpenFormula formulae can be converted with regular expressions.

Column Properties

The !cols array in each worksheet, if present, is a collection of ColInfo objects which have the following properties:

type ColInfo = {
  /* visibility */
  hidden?: boolean; // if true, the column is hidden

  /* column width is specified in one of the following ways: */
  wpx?:    number;  // width in screen pixels
  width?:  number;  // width in Excel's "Max Digit Width", width*256 is integral
  wch?:    number;  // width in characters

  /* other fields for preserving features from files */
  level?:  number;  // 0-indexed outline / group level
  MDW?:    number;  // Excel's "Max Digit Width" unit, always integral
};

Why are there three width types? (click to show)

There are three different width types corresponding to the three different ways spreadsheets store column widths:

SYLK and other plain text formats use raw character count. Contemporaneous tools like Visicalc and Multiplan were character based. Since the characters had the same width, it sufficed to store a count. This tradition was continued into the BIFF formats.

SpreadsheetML (2003) tried to align with HTML by standardizing on screen pixel count throughout the file. Column widths, row heights, and other measures use pixels. When the pixel and character counts do not align, Excel rounds values.

XLSX internally stores column widths in a nebulous "Max Digit Width" form. The Max Digit Width is the width of the largest digit when rendered (generally the "0" character is the widest). The internal width must be an integer multiple of the the width divided by 256. ECMA-376 describes a formula for converting between pixels and the internal width. This represents a hybrid approach.

Read functions attempt to populate all three properties. Write functions will try to cycle specified values to the desired type. In order to avoid potential conflicts, manipulation should delete the other properties first. For example, when changing the pixel width, delete the wch and width properties.

Implementation details (click to show)

Given the constraints, it is possible to determine the MDW without actually inspecting the font! The parsers guess the pixel width by converting from width to pixels and back, repeating for all possible MDW and selecting the MDW that minimizes the error. XLML actually stores the pixel width, so the guess works in the opposite direction.

Even though all of the information is made available, writers are expected to follow the priority order:

  1. use width field if available
  2. use wpx pixel width if available
  3. use wch character count if available

Row Properties

The !rows array in each worksheet, if present, is a collection of RowInfo objects which have the following properties:

type RowInfo = {
  /* visibility */
  hidden?: boolean; // if true, the row is hidden

  /* row height is specified in one of the following ways: */
  hpx?:    number;  // height in screen pixels
  hpt?:    number;  // height in points

  level?:  number;  // 0-indexed outline / group level
};

Note: Excel UI displays the base outline level as 1 and the max level as 8. The level field stores the base outline as 0 and the max level as 7.

Implementation details (click to show)

Excel internally stores row heights in points. The default resolution is 72 DPI or 96 PPI, so the pixel and point size should agree. For different resolutions they may not agree, so the library separates the concepts.

Even though all of the information is made available, writers are expected to follow the priority order:

  1. use hpx pixel height if available
  2. use hpt point height if available

Number Formats

The cell.w formatted text for each cell is produced from cell.v and cell.z format. If the format is not specified, the Excel General format is used. The format can either be specified as a string or as an index into the format table. Parsers are expected to populate workbook.SSF with the number format table. Writers are expected to serialize the table.

Custom tools should ensure that the local table has each used format string somewhere in the table. Excel convention mandates that the custom formats start at index 164. The following example creates a custom format from scratch:

New worksheet with custom format (click to show)

var wb = {
  SheetNames: ["Sheet1"],
  Sheets: {
    Sheet1: {
      "!ref":"A1:C1",
      A1: { t:"n", v:10000 },                    // <-- General format
      B1: { t:"n", v:10000, z: "0%" },           // <-- Builtin format
      C1: { t:"n", v:10000, z: "\"T\"\ #0.00" }  // <-- Custom format
    }
  }
}

The rules are slightly different from how Excel displays custom number formats. In particular, literal characters must be wrapped in double quotes or preceded by a backslash. For more info, see the Excel documentation article Create or delete a custom number format or ECMA-376 18.8.31 (Number Formats)

Default Number Formats (click to show)

The default formats are listed in ECMA-376 18.8.30:

IDFormat
0General
10
20.00
3#,##0
4#,##0.00
90%
100.00%
110.00E+00
12# ?/?
13# ??/??
14m/d/yy (see below)
15d-mmm-yy
16d-mmm
17mmm-yy
18h:mm AM/PM
19h:mm:ss AM/PM
20h:mm
21h:mm:ss
22m/d/yy h:mm
37#,##0 ;(#,##0)
38#,##0 ;[Red](#,##0)
39#,##0.00;(#,##0.00)
40#,##0.00;[Red](#,##0.00)
45mm:ss
46[h]:mm:ss
47mmss.0
48##0.0E+0
49@

Format 14 (m/d/yy) is localized by Excel: even though the file specifies that number format, it will be drawn differently based on system settings. It makes sense when the producer and consumer of files are in the same locale, but that is not always the case over the Internet. To get around this ambiguity, parse functions accept the dateNF option to override the interpretation of that specific format string.

Hyperlinks

Format Support (click to show)

Cell Hyperlinks: XLSX/M, XLSB, BIFF8 XLS, XLML, ODS

Tooltips: XLSX/M, XLSB, BIFF8 XLS, XLML

Hyperlinks are stored in the l key of cell objects. The Target field of the hyperlink object is the target of the link, including the URI fragment. Tooltips are stored in the Tooltip field and are displayed when you move your mouse over the text.

For example, the following snippet creates a link from cell A3 to https://sheetjs.com with the tip "Find us @ SheetJS.com!":

ws['A1'].l = { Target:"https://sheetjs.com", Tooltip:"Find us @ SheetJS.com!" };

Note that Excel does not automatically style hyperlinks -- they will generally be displayed as normal text.

Remote Links

HTTP / HTTPS links can be used directly:

ws['A2'].l = { Target:"https://docs.sheetjs.com/#hyperlinks" };
ws['A3'].l = { Target:"http://localhost:7262/yes_localhost_works" };

Excel also supports mailto email links with subject line:

ws['A4'].l = { Target:"mailto:ignored@dev.null" };
ws['A5'].l = { Target:"mailto:ignored@dev.null?subject=Test Subject" };

Local Links

Links to absolute paths should use the file:// URI scheme:

ws['B1'].l = { Target:"file:///SheetJS/t.xlsx" }; /* Link to /SheetJS/t.xlsx */
ws['B2'].l = { Target:"file:///c:/SheetJS.xlsx" }; /* Link to c:\SheetJS.xlsx */

Links to relative paths can be specified without a scheme:

ws['B3'].l = { Target:"SheetJS.xlsb" }; /* Link to SheetJS.xlsb */
ws['B4'].l = { Target:"../SheetJS.xlsm" }; /* Link to ../SheetJS.xlsm */

Relative Paths have undefined behavior in the SpreadsheetML 2003 format. Excel 2019 will treat a ..\ parent mark as two levels up.

Internal Links

Links where the target is a cell or range or defined name in the same workbook ("Internal Links") are marked with a leading hash character:

ws['C1'].l = { Target:"#E2" }; /* Link to cell E2 */
ws['C2'].l = { Target:"#Sheet2!E2" }; /* Link to cell E2 in sheet Sheet2 */
ws['C3'].l = { Target:"#SomeDefinedName" }; /* Link to Defined Name */

Cell Comments

Cell comments are objects stored in the c array of cell objects. The actual contents of the comment are split into blocks based on the comment author. The a field of each comment object is the author of the comment and the t field is the plain text representation.

For example, the following snippet appends a cell comment into cell A1:

if(!ws.A1.c) ws.A1.c = [];
ws.A1.c.push({a:"SheetJS", t:"I'm a little comment, short and stout!"});

Note: XLSB enforces a 54 character limit on the Author name. Names longer than 54 characters may cause issues with other formats.

To mark a comment as normally hidden, set the hidden property:

if(!ws.A1.c) ws.A1.c = [];
ws.A1.c.push({a:"SheetJS", t:"This comment is visible"});

if(!ws.A2.c) ws.A2.c = [];
ws.A2.c.hidden = true;
ws.A2.c.push({a:"SheetJS", t:"This comment will be hidden"});

Sheet Visibility

Excel enables hiding sheets in the lower tab bar. The sheet data is stored in the file but the UI does not readily make it available. Standard hidden sheets are revealed in the "Unhide" menu. Excel also has "very hidden" sheets which cannot be revealed in the menu. It is only accessible in the VB Editor!

The visibility setting is stored in the Hidden property of sheet props array.

More details (click to show)

ValueDefinition
0Visible
1Hidden
2Very Hidden

With https://rawgit.com/SheetJS/test_files/HEAD/sheet_visibility.xlsx:

> wb.Workbook.Sheets.map(function(x) { return [x.name, x.Hidden] })
[ [ 'Visible', 0 ], [ 'Hidden', 1 ], [ 'VeryHidden', 2 ] ]

Non-Excel formats do not support the Very Hidden state. The best way to test if a sheet is visible is to check if the Hidden property is logical truth:

> wb.Workbook.Sheets.map(function(x) { return [x.name, !x.Hidden] })
[ [ 'Visible', true ], [ 'Hidden', false ], [ 'VeryHidden', false ] ]

VBA and Macros

VBA Macros are stored in a special data blob that is exposed in the vbaraw property of the workbook object when the bookVBA option is true. They are supported in XLSM, XLSB, and BIFF8 XLS formats. The supported format writers automatically insert the data blobs if it is present in the workbook and associate with the worksheet names.

Custom Code Names (click to show)

The workbook code name is stored in wb.Workbook.WBProps.CodeName. By default, Excel will write ThisWorkbook or a translated phrase like DieseArbeitsmappe. Worksheet and Chartsheet code names are in the worksheet properties object at wb.Workbook.Sheets[i].CodeName. Macrosheets and Dialogsheets are ignored.

The readers and writers preserve the code names, but they have to be manually set when adding a VBA blob to a different workbook.

Macrosheets (click to show)

Older versions of Excel also supported a non-VBA "macrosheet" sheet type that stored automation commands. These are exposed in objects with the !type property set to "macro".

Detecting macros in workbooks (click to show)

The vbaraw field will only be set if macros are present, so testing is simple:

function wb_has_macro(wb/*:workbook*/)/*:boolean*/ {
    if(!!wb.vbaraw) return true;
    const sheets = wb.SheetNames.map((n) => wb.Sheets[n]);
    return sheets.some((ws) => !!ws && ws['!type']=='macro');
}

Parsing Options

The exported read and readFile functions accept an options argument:

Option NameDefaultDescription
type Input data encoding (see Input Type below)
rawfalseIf true, plain text parsing will not parse values **
codepage If specified, use code page when appropriate **
cellFormulatrueSave formulae to the .f field
cellHTMLtrueParse rich text and save HTML to the .h field
cellNFfalseSave number format string to the .z field
cellStylesfalseSave style/theme info to the .s field
cellTexttrueGenerated formatted text to the .w field
cellDatesfalseStore dates as type d (default is n)
dateNF If specified, use the string for date code 14 **
sheetStubsfalseCreate cell objects of type z for stub cells
sheetRows0If >0, read the first sheetRows rows **
bookDepsfalseIf true, parse calculation chains
bookFilesfalseIf true, add raw files to book object **
bookPropsfalseIf true, only parse enough to get book metadata **
bookSheetsfalseIf true, only parse enough to get the sheet names
bookVBAfalseIf true, copy VBA blob to vbaraw field **
password""If defined and file is encrypted, use password **
WTFfalseIf true, throw errors on unexpected file features **
sheets If specified, only parse specified sheets **
PRNfalseIf true, allow parsing of PRN files **
xlfnfalseIf true, preserve _xlfn. prefixes in formulae **
FS DSV Field Separator override
  • Even if cellNF is false, formatted text will be generated and saved to .w
  • In some cases, sheets may be parsed even if bookSheets is false.
  • Excel aggressively tries to interpret values from CSV and other plain text. This leads to surprising behavior! The raw option suppresses value parsing.
  • bookSheets and bookProps combine to give both sets of information
  • Deps will be an empty object if bookDeps is false
  • bookFiles behavior depends on file type:
    • keys array (paths in the ZIP) for ZIP-based formats
    • files hash (mapping paths to objects representing the files) for ZIP
    • cfb object for formats using CFB containers
  • sheetRows-1 rows will be generated when looking at the JSON object output (since the header row is counted as a row when parsing the data)
  • By default all worksheets are parsed. sheets restricts based on input type:
    • number: zero-based index of worksheet to parse (0 is first worksheet)
    • string: name of worksheet to parse (case insensitive)
    • array of numbers and strings to select multiple worksheets.
  • bookVBA merely exposes the raw VBA CFB object. It does not parse the data. XLSM and XLSB store the VBA CFB object in xl/vbaProject.bin. BIFF8 XLS mixes the VBA entries alongside the core Workbook entry, so the library generates a new XLSB-compatible blob from the XLS CFB container.
  • codepage is applied to BIFF2 - BIFF5 files without CodePage records and to CSV files without BOM in type:"binary". BIFF8 XLS always defaults to 1200.
  • PRN affects parsing of text files without a common delimiter character.
  • Currently only XOR encryption is supported. Unsupported error will be thrown for files employing other encryption methods.
  • Newer Excel functions are serialized with the _xlfn. prefix, hidden from the user. SheetJS will strip _xlfn. normally. The xlfn option preserves them.
  • WTF is mainly for development. By default, the parser will suppress read errors on single worksheets, allowing you to read from the worksheets that do parse properly. Setting WTF:true forces those errors to be thrown.

Input Type

Strings can be interpreted in multiple ways. The type parameter for read tells the library how to parse the data argument:

typeexpected input
"base64"string: Base64 encoding of the file
"binary"string: binary string (byte n is data.charCodeAt(n))
"string"string: JS string (characters interpreted as UTF8)
"buffer"nodejs Buffer
"array"array: array of 8-bit unsigned int (byte n is data[n])
"file"string: path of file that will be read (nodejs only)

Guessing File Type

Implementation Details (click to show)

Excel and other spreadsheet tools read the first few bytes and apply other heuristics to determine a file type. This enables file type punning: renaming files with the .xls extension will tell your computer to use Excel to open the file but Excel will know how to handle it. This library applies similar logic:

Byte 0Raw File TypeSpreadsheet Types
0xD0CFB ContainerBIFF 5/8 or protected XLSX/XLSB or WQ3/QPW or XLR
0x09BIFF StreamBIFF 2/3/4/5
0x3CXML/HTMLSpreadsheetML / Flat ODS / UOS1 / HTML / plain text
0x50ZIP ArchiveXLSB or XLSX/M or ODS or UOS2 or plain text
0x49Plain TextSYLK or plain text
0x54Plain TextDIF or plain text
0xEFUTF8 EncodedSpreadsheetML / Flat ODS / UOS1 / HTML / plain text
0xFFUTF16 EncodedSpreadsheetML / Flat ODS / UOS1 / HTML / plain text
0x00Record StreamLotus WK* or Quattro Pro or plain text
0x7BPlain textRTF or plain text
0x0APlain textSpreadsheetML / Flat ODS / UOS1 / HTML / plain text
0x0DPlain textSpreadsheetML / Flat ODS / UOS1 / HTML / plain text
0x20Plain textSpreadsheetML / Flat ODS / UOS1 / HTML / plain text

DBF files are detected based on the first byte as well as the third and fourth bytes (corresponding to month and day of the file date)

Works for Windows files are detected based on the BOF record with type 0xFF

Plain text format guessing follows the priority order:

FormatTest
XML<?xml appears in the first 1024 characters
HTMLstarts with < and HTML tags appear in the first 1024 characters *
XMLstarts with < and the first tag is valid
RTFstarts with {\rt
DSVstarts with /sep=.$/, separator is the specified character
DSVmore unquoted `
DSVmore unquoted ; chars than \t or , in the first 1024
TSVmore unquoted \t chars than , chars in the first 1024
CSVone of the first 1024 characters is a comma ","
ETHstarts with socialcalc:version:
PRNPRN option is set to true
CSV(fallback)
  • HTML tags include: html, table, head, meta, script, style, div

Why are random text files valid? (click to show)

Excel is extremely aggressive in reading files. Adding an XLS extension to any display text file (where the only characters are ANSI display chars) tricks Excel into thinking that the file is potentially a CSV or TSV file, even if it is only one column! This library attempts to replicate that behavior.

The best approach is to validate the desired worksheet and ensure it has the expected number of rows or columns. Extracting the range is extremely simple:

var range = XLSX.utils.decode_range(worksheet['!ref']);
var ncols = range.e.c - range.s.c + 1, nrows = range.e.r - range.s.r + 1;

Writing Options

The exported write and writeFile functions accept an options argument:

Option NameDefaultDescription
type Output data encoding (see Output Type below)
cellDatesfalseStore dates as type d (default is n)
bookSSTfalseGenerate Shared String Table **
bookType"xlsx"Type of Workbook (see below for supported formats)
sheet""Name of Worksheet for single-sheet formats **
compressionfalseUse ZIP compression for ZIP-based formats **
Props Override workbook properties when writing **
themeXLSX Override theme XML when writing XLSX/XLSB/XLSM **
ignoreECtrueSuppress "number as text" errors **
  • bookSST is slower and more memory intensive, but has better compatibility with older versions of iOS Numbers
  • The raw data is the only thing guaranteed to be saved. Features not described in this README may not be serialized.
  • cellDates only applies to XLSX output and is not guaranteed to work with third-party readers. Excel itself does not usually write cells with type d so non-Excel tools may ignore the data or error in the presence of dates.
  • Props is an object mirroring the workbook Props field. See the table from the Workbook File Properties section.
  • if specified, the string from themeXLSX will be saved as the primary theme for XLSX/XLSB/XLSM files (to xl/theme/theme1.xml in the ZIP)
  • Due to a bug in the program, some features like "Text to Columns" will crash Excel on worksheets where error conditions are ignored. The writer will mark files to ignore the error by default. Set ignoreEC to false to suppress.

Supported Output Formats

For broad compatibility with third-party tools, this library supports many output formats. The specific file type is controlled with bookType option:

bookTypefile extcontainersheetsDescription
xlsx.xlsxZIPmultiExcel 2007+ XML Format
xlsm.xlsmZIPmultiExcel 2007+ Macro XML Format
xlsb.xlsbZIPmultiExcel 2007+ Binary Format
biff8.xlsCFBmultiExcel 97-2004 Workbook Format
biff5.xlsCFBmultiExcel 5.0/95 Workbook Format
biff4.xlsnonesingleExcel 4.0 Worksheet Format
biff3.xlsnonesingleExcel 3.0 Worksheet Format
biff2.xlsnonesingleExcel 2.0 Worksheet Format
xlml.xlsnonemultiExcel 2003-2004 (SpreadsheetML)
ods.odsZIPmultiOpenDocument Spreadsheet
fods.fodsnonemultiFlat OpenDocument Spreadsheet
wk3.wk3nonesingleLotus Workbook (WK3)
csv.csvnonesingleComma Separated Values
txt.txtnonesingleUTF-16 Unicode Text (TXT)
sylk.sylknonesingleSymbolic Link (SYLK)
html.htmlnonesingleHTML Document
dif.difnonesingleData Interchange Format (DIF)
dbf.dbfnonesingledBASE II + VFP Extensions (DBF)
wk1.wk1nonesingleLotus Worksheet (WK1)
rtf.rtfnonesingleRich Text Format (RTF)
prn.prnnonesingleLotus Formatted Text
eth.ethnonesingleEthercalc Record Format (ETH)
  • compression only applies to formats with ZIP containers.
  • Formats that only support a single sheet require a sheet option specifying the worksheet. If the string is empty, the first worksheet is used.
  • writeFile will automatically guess the output file format based on the file extension if bookType is not specified. It will choose the first format in the aforementioned table that matches the extension.

Output Type

The type argument for write mirrors the type argument for read:

typeoutput
"base64"string: Base64 encoding of the file
"binary"string: binary string (byte n is data.charCodeAt(n))
"string"string: JS string (characters interpreted as UTF8)
"buffer"nodejs Buffer
"array"ArrayBuffer, fallback array of 8-bit unsigned int
"file"string: path of file that will be created (nodejs only)

Utility Functions

The sheet_to_* functions accept a worksheet and an optional options object.

The *_to_sheet functions accept a data object and an optional options object.

The examples are based on the following worksheet:

XXX| A | B | C | D | E | F | G |
---+---+---+---+---+---+---+---+
 1 | S | h | e | e | t | J | S |
 2 | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
 3 | 2 | 3 | 4 | 5 | 6 | 7 | 8 |

Array of Arrays Input

XLSX.utils.aoa_to_sheet takes an array of arrays of JS values and returns a worksheet resembling the input data. Numbers, Booleans and Strings are stored as the corresponding styles. Dates are stored as date or numbers. Array holes and explicit undefined values are skipped. null values may be stubbed. All other values are stored as strings. The function takes an options argument:

Option NameDefaultDescription
dateNFFMT 14Use specified date format in string output
cellDatesfalseStore dates as type d (default is n)
sheetStubsfalseCreate cell objects of type z for null values
nullErrorfalseIf true, emit #NULL! error cells for null values

Examples (click to show)

To generate the example sheet:

var ws = XLSX.utils.aoa_to_sheet([
  "SheetJS".split(""),
  [1,2,3,4,5,6,7],
  [2,3,4,5,6,7,8]
]);

XLSX.utils.sheet_add_aoa takes an array of arrays of JS values and updates an existing worksheet object. It follows the same process as aoa_to_sheet and accepts an options argument:

Option NameDefaultDescription
dateNFFMT 14Use specified date format in string output
cellDatesfalseStore dates as type d (default is n)
sheetStubsfalseCreate cell objects of type z for null values
nullErrorfalseIf true, emit #NULL! error cells for null values
origin Use specified cell as starting point (see below)

origin is expected to be one of:

originDescription
(cell object)Use specified cell (cell object)
(string)Use specified cell (A1-style cell)
(number >= 0)Start from the first column at specified row (0-indexed)
-1Append to bottom of worksheet starting on first column
(default)Start from cell A1

Examples (click to show)

Consider the worksheet:

XXX| A | B | C | D | E | F | G |
---+---+---+---+---+---+---+---+
 1 | S | h | e | e | t | J | S |
 2 | 1 | 2 |   |   | 5 | 6 | 7 |
 3 | 2 | 3 |   |   | 6 | 7 | 8 |
 4 | 3 | 4 |   |   | 7 | 8 | 9 |
 5 | 4 | 5 | 6 | 7 | 8 | 9 | 0 |

This worksheet can be built up in the order A1:G1, A2:B4, E2:G4, A5:G5:

/* Initial row */
var ws = XLSX.utils.aoa_to_sheet([ "SheetJS".split("") ]);

/* Write data starting at A2 */
XLSX.utils.sheet_add_aoa(ws, [[1,2], [2,3], [3,4]], {origin: "A2"});

/* Write data starting at E2 */
XLSX.utils.sheet_add_aoa(ws, [[5,6,7], [6,7,8], [7,8,9]], {origin:{r:1, c:4}});

/* Append row */
XLSX.utils.sheet_add_aoa(ws, [[4,5,6,7,8,9,0]], {origin: -1});

Array of Objects Input

XLSX.utils.json_to_sheet takes an array of objects and returns a worksheet with automatically-generated "headers" based on the keys of the objects. The default column order is determined by the first appearance of the field using Object.keys. The function accepts an options argument:

Option NameDefaultDescription
header Use specified field order (default Object.keys) **
dateNFFMT 14Use specified date format in string output
cellDatesfalseStore dates as type d (default is n)
skipHeaderfalseIf true, do not include header row in output
nullErrorfalseIf true, emit #NULL! error cells for null values
  • All fields from each row will be written. If header is an array and it does not contain a particular field, the key will be appended to the array.
  • Cell types are deduced from the type of each value. For example, a Date object will generate a Date cell, while a string will generate a Text cell.
  • Null values will be skipped by default. If nullError is true, an error cell corresponding to #NULL! will be written to the worksheet.

Examples (click to show)

The original sheet cannot be reproduced using plain objects since JS object keys must be unique. After replacing the second e and S with e_1 and S_1:

var ws = XLSX.utils.json_to_sheet([
  { S:1, h:2, e:3, e_1:4, t:5, J:6, S_1:7 },
  { S:2, h:3, e:4, e_1:5, t:6, J:7, S_1:8 }
], {header:["S","h","e","e_1","t","J","S_1"]});

Alternatively, the header row can be skipped:

var ws = XLSX.utils.json_to_sheet([
  { A:"S", B:"h", C:"e", D:"e", E:"t", F:"J", G:"S" },
  { A: 1,  B: 2,  C: 3,  D: 4,  E: 5,  F: 6,  G: 7  },
  { A: 2,  B: 3,  C: 4,  D: 5,  E: 6,  F: 7,  G: 8  }
], {header:["A","B","C","D","E","F","G"], skipHeader:true});

XLSX.utils.sheet_add_json takes an array of objects and updates an existing worksheet object. It follows the same process as json_to_sheet and accepts an options argument:

Option NameDefaultDescription
header Use specified column order (default Object.keys)
dateNFFMT 14Use specified date format in string output
cellDatesfalseStore dates as type d (default is n)
skipHeaderfalseIf true, do not include header row in output
nullErrorfalseIf true, emit #NULL! error cells for null values
origin Use specified cell as starting point (see below)

origin is expected to be one of:

originDescription
(cell object)Use specified cell (cell object)
(string)Use specified cell (A1-style cell)
(number >= 0)Start from the first column at specified row (0-indexed)
-1Append to bottom of worksheet starting on first column
(default)Start from cell A1

Examples (click to show)

Consider the worksheet:

XXX| A | B | C | D | E | F | G |
---+---+---+---+---+---+---+---+
 1 | S | h | e | e | t | J | S |
 2 | 1 | 2 |   |   | 5 | 6 | 7 |
 3 | 2 | 3 |   |   | 6 | 7 | 8 |
 4 | 3 | 4 |   |   | 7 | 8 | 9 |
 5 | 4 | 5 | 6 | 7 | 8 | 9 | 0 |

This worksheet can be built up in the order A1:G1, A2:B4, E2:G4, A5:G5:

/* Initial row */
var ws = XLSX.utils.json_to_sheet([
  { A: "S", B: "h", C: "e", D: "e", E: "t", F: "J", G: "S" }
], {header: ["A", "B", "C", "D", "E", "F", "G"], skipHeader: true});

/* Write data starting at A2 */
XLSX.utils.sheet_add_json(ws, [
  { A: 1, B: 2 }, { A: 2, B: 3 }, { A: 3, B: 4 }
], {skipHeader: true, origin: "A2"});

/* Write data starting at E2 */
XLSX.utils.sheet_add_json(ws, [
  { A: 5, B: 6, C: 7 }, { A: 6, B: 7, C: 8 }, { A: 7, B: 8, C: 9 }
], {skipHeader: true, origin: { r: 1, c: 4 }, header: [ "A", "B", "C" ]});

/* Append row */
XLSX.utils.sheet_add_json(ws, [
  { A: 4, B: 5, C: 6, D: 7, E: 8, F: 9, G: 0 }
], {header: ["A", "B", "C", "D", "E", "F", "G"], skipHeader: true, origin: -1});

HTML Table Input

XLSX.utils.table_to_sheet takes a table DOM element and returns a worksheet resembling the input table. Numbers are parsed. All other data will be stored as strings.

XLSX.utils.table_to_book produces a minimal workbook based on the worksheet.

Both functions accept options arguments:

Option NameDefaultDescription
raw If true, every cell will hold raw strings
dateNFFMT 14Use specified date format in string output
cellDatesfalseStore dates as type d (default is n)
sheetRows0If >0, read the first sheetRows rows of the table
displayfalseIf true, hidden rows and cells will not be parsed

Examples (click to show)

To generate the example sheet, start with the HTML table:

<table id="sheetjs">
<tr><td>S</td><td>h</td><td>e</td><td>e</td><td>t</td><td>J</td><td>S</td></tr>
<tr><td>1</td><td>2</td><td>3</td><td>4</td><td>5</td><td>6</td><td>7</td></tr>
<tr><td>2</td><td>3</td><td>4</td><td>5</td><td>6</td><td>7</td><td>8</td></tr>
</table>

To process the table:

var tbl = document.getElementById('sheetjs');
var wb = XLSX.utils.table_to_book(tbl);

Note: XLSX.read can handle HTML represented as strings.

XLSX.utils.sheet_add_dom takes a table DOM element and updates an existing worksheet object. It follows the same process as table_to_sheet and accepts an options argument:

Option NameDefaultDescription
raw If true, every cell will hold raw strings
dateNFFMT 14Use specified date format in string output
cellDatesfalseStore dates as type d (default is n)
sheetRows0If >0, read the first sheetRows rows of the table
displayfalseIf true, hidden rows and cells will not be parsed

origin is expected to be one of:

originDescription
(cell object)Use specified cell (cell object)
(string)Use specified cell (A1-style cell)
(number >= 0)Start from the first column at specified row (0-indexed)
-1Append to bottom of worksheet starting on first column
(default)Start from cell A1

Examples (click to show)

A small helper function can create gap rows between tables:

function create_gap_rows(ws, nrows) {
  var ref = XLSX.utils.decode_range(ws["!ref"]);       // get original range
  ref.e.r += nrows;                                    // add to ending row
  ws["!ref"] = XLSX.utils.encode_range(ref);           // reassign row
}

/* first table */
var ws = XLSX.utils.table_to_sheet(document.getElementById('table1'));
create_gap_rows(ws, 1); // one row gap after first table

/* second table */
XLSX.utils.sheet_add_dom(ws, document.getElementById('table2'), {origin: -1});
create_gap_rows(ws, 3); // three rows gap after second table

/* third table */
XLSX.utils.sheet_add_dom(ws, document.getElementById('table3'), {origin: -1});

Formulae Output

XLSX.utils.sheet_to_formulae generates an array of commands that represent how a person would enter data into an application. Each entry is of the form A1-cell-address=formula-or-value. String literals are prefixed with a ' in accordance with Excel.

Examples (click to show)

For the example sheet:

> var o = XLSX.utils.sheet_to_formulae(ws);
> [o[0], o[5], o[10], o[15], o[20]];
[ 'A1=\'S', 'F1=\'J', 'D2=4', 'B3=3', 'G3=8' ]

Delimiter-Separated Output

As an alternative to the writeFile CSV type, XLSX.utils.sheet_to_csv also produces CSV output. The function takes an options argument:

Option NameDefaultDescription
FS",""Field Separator" delimiter between fields
RS"\n""Record Separator" delimiter between rows
dateNFFMT 14Use specified date format in string output
stripfalseRemove trailing field separators in each record **
blankrowstrueInclude blank lines in the CSV output
skipHiddenfalseSkips hidden rows/columns in the CSV output
forceQuotesfalseForce quotes around fields
  • strip will remove trailing commas from each line under default FS/RS
  • blankrows must be set to false to skip blank lines.
  • Fields containing the record or field separator will automatically be wrapped in double quotes; forceQuotes forces all cells to be wrapped in quotes.

Examples (click to show)

For the example sheet:

> console.log(XLSX.utils.sheet_to_csv(ws));
S,h,e,e,t,J,S
1,2,3,4,5,6,7
2,3,4,5,6,7,8
> console.log(XLSX.utils.sheet_to_csv(ws, {FS:"\t"}));
S    h    e    e    t    J    S
1    2    3    4    5    6    7
2    3    4    5    6    7    8
> console.log(XLSX.utils.sheet_to_csv(ws,{FS:":",RS:"|"}));
S:h:e:e:t:J:S|1:2:3:4:5:6:7|2:3:4:5:6:7:8|

UTF-16 Unicode Text

The txt output type uses the tab character as the field separator. If the codepage library is available (included in full distribution but not core), the output will be encoded in CP1200 and the BOM will be prepended.

XLSX.utils.sheet_to_txt takes the same arguments as sheet_to_csv.

HTML Output

As an alternative to the writeFile HTML type, XLSX.utils.sheet_to_html also produces HTML output. The function takes an options argument:

Option NameDefaultDescription
id Specify the id attribute for the TABLE element
editablefalseIf true, set contenteditable="true" for every TD
header Override header (default html body)
footer Override footer (default /body /html)

Examples (click to show)

For the example sheet:

> console.log(XLSX.utils.sheet_to_html(ws));
// ...

JSON

XLSX.utils.sheet_to_json generates different types of JS objects. The function takes an options argument:

Option NameDefaultDescription
rawtrueUse raw values (true) or formatted strings (false)
rangefrom WSOverride Range (see table below)
header Control output format (see table below)
dateNFFMT 14Use specified date format in string output
defval Use specified value in place of null or undefined
blankrows**Include blank lines in the output **
  • raw only affects cells which have a format code (.z) field or a formatted text (.w) field.
  • If header is specified, the first row is considered a data row; if header is not specified, the first row is the header row and not considered data.
  • When header is not specified, the conversion will automatically disambiguate header entries by affixing _ and a count starting at 1. For example, if three columns have header foo the output fields are foo, foo_1, foo_2
  • null values are returned when raw is true but are skipped when false.
  • If defval is not specified, null and undefined values are skipped normally. If specified, all null and undefined points will be filled with defval
  • When header is 1, the default is to generate blank rows. blankrows must be set to false to skip blank rows.
  • When header is not 1, the default is to skip blank rows. blankrows must be true to generate blank rows

range is expected to be one of:

rangeDescription
(number)Use worksheet range but set starting row to the value
(string)Use specified range (A1-style bounded range string)
(default)Use worksheet range (ws['!ref'])

header is expected to be one of:

headerDescription
1Generate an array of arrays ("2D Array")
"A"Row object keys are literal column labels
array of stringsUse specified strings as keys in row objects
(default)Read and disambiguate first row as keys

If header is not 1, the row object will contain the non-enumerable property __rowNum__ that represents the row of the sheet corresponding to the entry.

Examples (click to show)

For the example sheet:

> XLSX.utils.sheet_to_json(ws);
[ { S: 1, h: 2, e: 3, e_1: 4, t: 5, J: 6, S_1: 7 },
  { S: 2, h: 3, e: 4, e_1: 5, t: 6, J: 7, S_1: 8 } ]

> XLSX.utils.sheet_to_json(ws, {header:"A"});
[ { A: 'S', B: 'h', C: 'e', D: 'e', E: 't', F: 'J', G: 'S' },
  { A: '1', B: '2', C: '3', D: '4', E: '5', F: '6', G: '7' },
  { A: '2', B: '3', C: '4', D: '5', E: '6', F: '7', G: '8' } ]

> XLSX.utils.sheet_to_json(ws, {header:["A","E","I","O","U","6","9"]});
[ { '6': 'J', '9': 'S', A: 'S', E: 'h', I: 'e', O: 'e', U: 't' },
  { '6': '6', '9': '7', A: '1', E: '2', I: '3', O: '4', U: '5' },
  { '6': '7', '9': '8', A: '2', E: '3', I: '4', O: '5', U: '6' } ]

> XLSX.utils.sheet_to_json(ws, {header:1});
[ [ 'S', 'h', 'e', 'e', 't', 'J', 'S' ],
  [ '1', '2', '3', '4', '5', '6', '7' ],
  [ '2', '3', '4', '5', '6', '7', '8' ] ]

Example showing the effect of raw:

> ws['A2'].w = "3";                          // set A2 formatted string value

> XLSX.utils.sheet_to_json(ws, {header:1, raw:false});
[ [ 'S', 'h', 'e', 'e', 't', 'J', 'S' ],
  [ '3', '2', '3', '4', '5', '6', '7' ],     // <-- A2 uses the formatted string
  [ '2', '3', '4', '5', '6', '7', '8' ] ]

> XLSX.utils.sheet_to_json(ws, {header:1});
[ [ 'S', 'h', 'e', 'e', 't', 'J', 'S' ],
  [ 1, 2, 3, 4, 5, 6, 7 ],                   // <-- A2 uses the raw value
  [ 2, 3, 4, 5, 6, 7, 8 ] ]

File Formats

Despite the library name xlsx, it supports numerous spreadsheet file formats:

FormatReadWrite
Excel Worksheet/Workbook Formats:-----::-----:
Excel 2007+ XML Formats (XLSX/XLSM)
Excel 2007+ Binary Format (XLSB BIFF12)
Excel 2003-2004 XML Format (XML "SpreadsheetML")
Excel 97-2004 (XLS BIFF8)
Excel 5.0/95 (XLS BIFF5)
Excel 4.0 (XLS/XLW BIFF4)
Excel 3.0 (XLS BIFF3)
Excel 2.0/2.1 (XLS BIFF2)
Excel Supported Text Formats:-----::-----:
Delimiter-Separated Values (CSV/TXT)
Data Interchange Format (DIF)
Symbolic Link (SYLK/SLK)
Lotus Formatted Text (PRN)
UTF-16 Unicode Text (TXT)
Other Workbook/Worksheet Formats:-----::-----:
OpenDocument Spreadsheet (ODS)
Flat XML ODF Spreadsheet (FODS)
Uniform Office Format Spreadsheet (标文通 UOS1/UOS2) 
dBASE II/III/IV / Visual FoxPro (DBF)
Lotus 1-2-3 (WK1/WK3)
Lotus 1-2-3 (WKS/WK2/WK4/123) 
Quattro Pro Spreadsheet (WQ1/WQ2/WB1/WB2/WB3/QPW) 
Works 1.x-3.x DOS / 2.x-5.x Windows Spreadsheet (WKS) 
Works 6.x-9.x Spreadsheet (XLR) 
Other Common Spreadsheet Output Formats:-----::-----:
HTML Tables
Rich Text Format tables (RTF) 
Ethercalc Record Format (ETH)

Features not supported by a given file format will not be written. Formats with range limits will be silently truncated:

FormatLast CellMax ColsMax Rows
Excel 2007+ XML Formats (XLSX/XLSM)XFD1048576163841048576
Excel 2007+ Binary Format (XLSB BIFF12)XFD1048576163841048576
Excel 97-2004 (XLS BIFF8)IV6553625665536
Excel 5.0/95 (XLS BIFF5)IV1638425616384
Excel 4.0 (XLS BIFF4)IV1638425616384
Excel 3.0 (XLS BIFF3)IV1638425616384
Excel 2.0/2.1 (XLS BIFF2)IV1638425616384
Lotus 1-2-3 R2 - R5 (WK1/WK3/WK4)IV81922568192
Lotus 1-2-3 R1 (WKS)IV20482562048

Excel 2003 SpreadsheetML range limits are governed by the version of Excel and are not enforced by the writer.

Excel 2007+ XML (XLSX/XLSM)

(click to show)

XLSX and XLSM files are ZIP containers containing a series of XML files in accordance with the Open Packaging Conventions (OPC). The XLSM format, almost identical to XLSX, is used for files containing macros.

The format is standardized in ECMA-376 and later in ISO/IEC 29500. Excel does not follow the specification, and there are additional documents discussing how Excel deviates from the specification.

Excel 2.0-95 (BIFF2/BIFF3/BIFF4/BIFF5)

(click to show)

BIFF 2/3 XLS are single-sheet streams of binary records. Excel 4 introduced the concept of a workbook (XLW files) but also had single-sheet XLS format. The structure is largely similar to the Lotus 1-2-3 file formats. BIFF5/8/12 extended the format in various ways but largely stuck to the same record format.

There is no official specification for any of these formats. Excel 95 can write files in these formats, so record lengths and fields were determined by writing in all of the supported formats and comparing files. Excel 2016 can generate BIFF5 files, enabling a full suite of file tests starting from XLSX or BIFF2.

Excel 97-2004 Binary (BIFF8)

(click to show)

BIFF8 exclusively uses the Compound File Binary container format, splitting some content into streams within the file. At its core, it still uses an extended version of the binary record format from older versions of BIFF.

The MS-XLS specification covers the basics of the file format, and other specifications expand on serialization of features like properties.

Excel 2003-2004 (SpreadsheetML)

(click to show)

Predating XLSX, SpreadsheetML files are simple XML files. There is no official and comprehensive specification, although MS has released documentation on the format. Since Excel 2016 can generate SpreadsheetML files, mapping features is pretty straightforward.

Excel 2007+ Binary (XLSB, BIFF12)

(click to show)

Introduced in parallel with XLSX, the XLSB format combines the BIFF architecture with the content separation and ZIP container of XLSX. For the most part nodes in an XLSX sub-file can be mapped to XLSB records in a corresponding sub-file.

The MS-XLSB specification covers the basics of the file format, and other specifications expand on serialization of features like properties.

Delimiter-Separated Values (CSV/TXT)

(click to show)

Excel CSV deviates from RFC4180 in a number of important ways. The generated CSV files should generally work in Excel although they may not work in RFC4180 compatible readers. The parser should generally understand Excel CSV. The writer proactively generates cells for formulae if values are unavailable.

Excel TXT uses tab as the delimiter and code page 1200.

Notes:

  • Like in Excel, files starting with 0x49 0x44 ("ID") are treated as Symbolic Link files. Unlike Excel, if the file does not have a valid SYLK header, it will be proactively reinterpreted as CSV. There are some files with semicolon delimiter that align with a valid SYLK file. For the broadest compatibility, all cells with the value of ID are automatically wrapped in double-quotes.

Other Workbook Formats

(click to show)

Support for other formats is generally far XLS/XLSB/XLSX support, due in large part to a lack of publicly available documentation. Test files were produced in the respective apps and compared to their XLS exports to determine structure. The main focus is data extraction.

Lotus 1-2-3 (WKS/WK1/WK2/WK3/WK4/123)

(click to show)

The Lotus formats consist of binary records similar to the BIFF structure. Lotus did release a specification decades ago covering the original WK1 format. Other features were deduced by producing files and comparing to Excel support.

Generated WK1 worksheets are compatible with Lotus 1-2-3 R2 and Excel 5.0.

Generated WK3 workbooks are compatible with Lotus 1-2-3 R9 and Excel 5.0.

Quattro Pro (WQ1/WQ2/WB1/WB2/WB3/QPW)

(click to show)

The Quattro Pro formats use binary records in the same way as BIFF and Lotus. Some of the newer formats (namely WB3 and QPW) use a CFB enclosure just like BIFF8 XLS.

Works for DOS / Windows Spreadsheet (WKS/XLR)

(click to show)

All versions of Works were limited to a single worksheet.

Works for DOS 1.x - 3.x and Works for Windows 2.x extends the Lotus WKS format with additional record types.

Works for Windows 3.x - 5.x uses the same format and WKS extension. The BOF record has type FF

Works for Windows 6.x - 9.x use the XLR format. XLR is nearly identical to BIFF8 XLS: it uses the CFB container with a Workbook stream. Works 9 saves the exact Workbook stream for the XLR and the 97-2003 XLS export. Works 6 XLS includes two empty worksheets but the main worksheet has an identical encoding. XLR also includes a WksSSWorkBook stream similar to Lotus FM3/FMT files.

OpenDocument Spreadsheet (ODS/FODS)

(click to show)

ODS is an XML-in-ZIP format akin to XLSX while FODS is an XML format akin to SpreadsheetML. Both are detailed in the OASIS standard, but tools like LO/OO add undocumented extensions. The parsers and writers do not implement the full standard, instead focusing on parts necessary to extract and store raw data.

Uniform Office Spreadsheet (UOS1/2)

(click to show)

UOS is a very similar format, and it comes in 2 varieties corresponding to ODS and FODS respectively. For the most part, the difference between the formats is in the names of tags and attributes.

Other Single-Worksheet Formats

Many older formats supported only one worksheet:

dBASE and Visual FoxPro (DBF)

(click to show)

DBF is really a typed table format: each column can only hold one data type and each record omits type information. The parser generates a header row and inserts records starting at the second row of the worksheet. The writer makes files compatible with Visual FoxPro extensions.

Multi-file extensions like external memos and tables are currently unsupported, limited by the general ability to read arbitrary files in the web browser. The reader understands DBF Level 7 extensions like DATETIME.

Symbolic Link (SYLK)

(click to show)

There is no real documentation. All knowledge was gathered by saving files in various versions of Excel to deduce the meaning of fields. Notes:

  • Plain formulae are stored in the RC form.
  • Column widths are rounded to integral characters.

Lotus Formatted Text (PRN)

(click to show)

There is no real documentation, and in fact Excel treats PRN as an output-only file format. Nevertheless we can guess the column widths and reverse-engineer the original layout. Excel's 240 character width limitation is not enforced.

Data Interchange Format (DIF)

(click to show)

There is no unified definition. Visicalc DIF differs from Lotus DIF, and both differ from Excel DIF. Where ambiguous, the parser/writer follows the expected behavior from Excel. In particular, Excel extends DIF in incompatible ways:

  • Since Excel automatically converts numbers-as-strings to numbers, numeric string constants are converted to formulae: "0.3" -> "=""0.3""
  • DIF technically expects numeric cells to hold the raw numeric data, but Excel permits formatted numbers (including dates)
  • DIF technically has no support for formulae, but Excel will automatically convert plain formulae. Array formulae are not preserved.

HTML

(click to show)

Excel HTML worksheets include special metadata encoded in styles. For example, mso-number-format is a localized string containing the number format. Despite the metadata the output is valid HTML, although it does accept bare & symbols.

The writer adds type metadata to the TD elements via the t tag. The parser looks for those tags and overrides the default interpretation. For example, text like <td>12345</td> will be parsed as numbers but <td t="s">12345</td> will be parsed as text.

Rich Text Format (RTF)

(click to show)

Excel RTF worksheets are stored in clipboard when copying cells or ranges from a worksheet. The supported codes are a subset of the Word RTF support.

Ethercalc Record Format (ETH)

(click to show)

Ethercalc is an open source web spreadsheet powered by a record format reminiscent of SYLK wrapped in a MIME multi-part message.

Testing

Node

(click to show)

make test will run the node-based tests. By default it runs tests on files in every supported format. To test a specific file type, set FMTS to the format you want to test. Feature-specific tests are available with make test_misc

$ make test_misc   # run core tests
$ make test        # run full tests
$ make test_xls    # only use the XLS test files
$ make test_xlsx   # only use the XLSX test files
$ make test_xlsb   # only use the XLSB test files
$ make test_xml    # only use the XML test files
$ make test_ods    # only use the ODS test files

To enable all errors, set the environment variable WTF=1:

$ make test        # run full tests
$ WTF=1 make test  # enable all error messages

flow and eslint checks are available:

$ make lint        # eslint checks
$ make flow        # make lint + Flow checking
$ make tslint      # check TS definitions

Browser

(click to show)

The core in-browser tests are available at tests/index.html within this repo. Start a local server and navigate to that directory to run the tests. make ctestserv will start a server on port 8000.

make ctest will generate the browser fixtures. To add more files, edit the tests/fixtures.lst file and add the paths.

To run the full in-browser tests, clone the repo for oss.sheetjs.com and replace the xlsx.js file (then open a browser window and go to stress.html):

$ cp xlsx.js ../SheetJS.github.io
$ cd ../SheetJS.github.io
$ simplehttpserver # or "python -mSimpleHTTPServer" or "serve"
$ open -a Chromium.app http://localhost:8000/stress.html

Tested Environments

(click to show)

  • NodeJS 0.8, 0.10, 0.12, 4.x, 5.x, 6.x, 7.x, 8.x
  • IE 6/7/8/9/10/11 (IE 6-9 require shims)
  • Chrome 24+ (including Android 4.0+)
  • Safari 6+ (iOS and Desktop)
  • Edge 13+, FF 18+, and Opera 12+

Tests utilize the mocha testing framework.

The test suite also includes tests for various time zones. To change the timezone locally, set the TZ environment variable:

$ env TZ="Asia/Kolkata" WTF=1 make test_misc

Test Files

Test files are housed in another repo.

Running make init will refresh the test_files submodule and get the files. Note that this requires svn, git, hg and other commands that may not be available. If make init fails, please download the latest version of the test files snapshot from the repo

Latest Snapshot (click to show)

Latest test files snapshot: http://github.com/SheetJS/test_files/releases/download/20170409/test_files.zip

(download and unzip to the test_files subdirectory)

Contributing

Due to the precarious nature of the Open Specifications Promise, it is very important to ensure code is cleanroom. Contribution Notes

File organization (click to show)

At a high level, the final script is a concatenation of the individual files in the bits folder. Running make should reproduce the final output on all platforms. The README is similarly split into bits in the docbits folder.

Folders:

foldercontents
bitsraw source files that make up the final script
docbitsraw markdown files that make up README.md
binserver-side bin scripts (xlsx.njs)
distdist files for web browsers and nonstandard JS environments
demosdemo projects for platforms like ExtendScript and Webpack
testsbrowser tests (run make ctest to rebuild)
typestypescript definitions and tests
miscmiscellaneous supporting scripts
test_filestest files (pulled from the test files repository)

After cloning the repo, running make help will display a list of commands.

OSX/Linux

(click to show)

The xlsx.js file is constructed from the files in the bits subdirectory. The build script (run make) will concatenate the individual bits to produce the script. Before submitting a contribution, ensure that running make will produce the xlsx.js file exactly. The simplest way to test is to add the script:

$ git add xlsx.js
$ make clean
$ make
$ git diff xlsx.js

To produce the dist files, run make dist. The dist files are updated in each version release and should not be committed between versions.

Windows

(click to show)

The included make.cmd script will build xlsx.js from the bits directory. Building is as simple as:

> make

To prepare development environment:

> make init

The full list of commands available in Windows are displayed in make help:

make init -- install deps and global modules
make lint -- run eslint linter
make test -- run mocha test suite
make misc -- run smaller test suite
make book -- rebuild README and summary
make help -- display this message

As explained in Test Files, on Windows the release ZIP file must be downloaded and extracted. If Bash on Windows is available, it is possible to run the OSX/Linux workflow. The following steps prepares the environment:

# Install support programs for the build and test commands
sudo apt-get install make git subversion mercurial

# Install nodejs and NPM within the WSL
wget -qO- https://deb.nodesource.com/setup_8.x | sudo bash
sudo apt-get install nodejs

# Install dev dependencies
sudo npm install -g mocha voc blanket xlsjs

Tests

(click to show)

The test_misc target (make test_misc on Linux/OSX / make misc on Windows) runs the targeted feature tests. It should take 5-10 seconds to perform feature tests without testing against the entire test battery. New features should be accompanied with tests for the relevant file formats and features.

For tests involving the read side, an appropriate feature test would involve reading an existing file and checking the resulting workbook object. If a parameter is involved, files should be read with different values to verify that the feature is working as expected.

For tests involving a new write feature which can already be parsed, appropriate feature tests would involve writing a workbook with the feature and then opening and verifying that the feature is preserved.

For tests involving a new write feature without an existing read ability, please add a feature test to the kitchen sink tests/write.js.

License

Please consult the attached LICENSE file for details. All rights not explicitly granted by the Apache 2.0 License are reserved by the Original Author.

References

OSP-covered Specifications (click to show)

  • MS-CFB: Compound File Binary File Format
  • MS-CTXLS: Excel Custom Toolbar Binary File Format
  • MS-EXSPXML3: Excel Calculation Version 2 Web Service XML Schema
  • MS-ODATA: Open Data Protocol (OData)
  • MS-ODRAW: Office Drawing Binary File Format
  • MS-ODRAWXML: Office Drawing Extensions to Office Open XML Structure
  • MS-OE376: Office Implementation Information for ECMA-376 Standards Support
  • MS-OFFCRYPTO: Office Document Cryptography Structure
  • MS-OI29500: Office Implementation Information for ISO/IEC 29500 Standards Support
  • MS-OLEDS: Object Linking and Embedding (OLE) Data Structures
  • MS-OLEPS: Object Linking and Embedding (OLE) Property Set Data Structures
  • MS-OODF3: Office Implementation Information for ODF 1.2 Standards Support
  • MS-OSHARED: Office Common Data Types and Objects Structures
  • MS-OVBA: Office VBA File Format Structure
  • MS-XLDM: Spreadsheet Data Model File Format
  • MS-XLS: Excel Binary File Format (.xls) Structure Specification
  • MS-XLSB: Excel (.xlsb) Binary File Format
  • MS-XLSX: Excel (.xlsx) Extensions to the Office Open XML SpreadsheetML File Format
  • XLS: Microsoft Office Excel 97-2007 Binary File Format Specification
  • RTF: Rich Text Format
  • ISO/IEC 29500:2012(E) "Information technology — Document description and processing languages — Office Open XML File Formats"
  • Open Document Format for Office Applications Version 1.2 (29 September 2011)
  • Worksheet File Format (From Lotus) December 1984

Download Details:
Author: SheetJS
Source Code: https://github.com/SheetJS/sheetjs
License: Apache-2.0 License

#database  #javascript #react 

Brain  Crist

Brain Crist

1603659600

Dart sound null safety: technical preview 2

Dart serves a special role in Flutter, powering developer features such as hot reload, and enabling multi-platform apps for mobile, desktop, and web via Dart’s flexible compiler technology. We strive to make the Dart language the most productive for Flutter app developers; for example, we added UI-as-code language constructs to optimize the Dart syntax for coding Flutter widget trees.

In June we offered a first tech preview of null safety for Dart. Today is another major milestone, which we’ve looked forward to for a while: We’re announcing a second tech preview of sound null safety, including support for the Flutter framework.

Null safety is a major new productivity feature that helps you avoid null exceptions, a class of bugs that are often hard to spot. As an added bonus, this feature also enables a range of performance improvements. We’re really looking forward to your feedback.

Why null safety?

Dart is a type-safe language. This means that when you get a variable of some type, the compiler can guarantee that it is of that type. But type safety by itself doesn’t guarantee that the variable is not null.

Null errors are very common. A search on GitHub leads to thousands of issues caused by unexpected nulls in Dart code, and even more thousands of commits trying to fix those issues. Try to see if you can spot the nullability problems in the following Flutter app, imagining that Config and WeatherService are backend services used by the app:

This app will certainly fail if getAppName() returns a null; in that case we’ll pass a null to the Text widget used in the title of AppBar.

But there are more subtle cases to consider: getTemperatures() could also return null. In that case the for-loop will fail. Or getTemperatures() could return a list as expected, but that list might contain null values, in which case we’ll call round() on null, and the app will fail.

The null safety feature makes these problems go away by validating your code as you’re typing:

Screenshot of the preceding code with null errors.

With null safety, Dart finds potential null errors in your code.

With null safety, you can reason about your code with more confidence. No more pesky runtime null dereferencing errors in deployed apps. Instead, you get static errors as you code.

#programming #dart #null-safety #announcements #flutter

A Wrapper for Sembast and SQFlite to Enable Easy

FHIR_DB

This is really just a wrapper around Sembast_SQFLite - so all of the heavy lifting was done by Alex Tekartik. I highly recommend that if you have any questions about working with this package that you take a look at Sembast. He's also just a super nice guy, and even answered a question for me when I was deciding which sembast version to use. As usual, ResoCoder also has a good tutorial.

I have an interest in low-resource settings and thus a specific reason to be able to store data offline. To encourage this use, there are a number of other packages I have created based around the data format FHIR. FHIR® is the registered trademark of HL7 and is used with the permission of HL7. Use of the FHIR trademark does not constitute endorsement of this product by HL7.

Using the Db

So, while not absolutely necessary, I highly recommend that you use some sort of interface class. This adds the benefit of more easily handling errors, plus if you change to a different database in the future, you don't have to change the rest of your app, just the interface.

I've used something like this in my projects:

class IFhirDb {
  IFhirDb();
  final ResourceDao resourceDao = ResourceDao();

  Future<Either<DbFailure, Resource>> save(Resource resource) async {
    Resource resultResource;
    try {
      resultResource = await resourceDao.save(resource);
    } catch (error) {
      return left(DbFailure.unableToSave(error: error.toString()));
    }
    return right(resultResource);
  }

  Future<Either<DbFailure, List<Resource>>> returnListOfSingleResourceType(
      String resourceType) async {
    List<Resource> resultList;
    try {
      resultList =
          await resourceDao.getAllSortedById(resourceType: resourceType);
    } catch (error) {
      return left(DbFailure.unableToObtainList(error: error.toString()));
    }
    return right(resultList);
  }

  Future<Either<DbFailure, List<Resource>>> searchFunction(
      String resourceType, String searchString, String reference) async {
    List<Resource> resultList;
    try {
      resultList =
          await resourceDao.searchFor(resourceType, searchString, reference);
    } catch (error) {
      return left(DbFailure.unableToObtainList(error: error.toString()));
    }
    return right(resultList);
  }
}

I like this because in case there's an i/o error or something, it won't crash your app. Then, you can call this interface in your app like the following:

final patient = Patient(
    resourceType: 'Patient',
    name: [HumanName(text: 'New Patient Name')],
    birthDate: Date(DateTime.now()),
);

final saveResult = await IFhirDb().save(patient);

This will save your newly created patient to the locally embedded database.

IMPORTANT: this database will expect that all previously created resources have an id. When you save a resource, it will check to see if that resource type has already been stored. (Each resource type is saved in it's own store in the database). It will then check if there is an ID. If there's no ID, it will create a new one for that resource (along with metadata on version number and creation time). It will save it, and return the resource. If it already has an ID, it will copy the the old version of the resource into a _history store. It will then update the metadata of the new resource and save that version into the appropriate store for that resource. If, for instance, we have a previously created patient:

{
    "resourceType": "Patient",
    "id": "fhirfli-294057507-6811107",
    "meta": {
        "versionId": "1",
        "lastUpdated": "2020-10-16T19:41:28.054369Z"
    },
    "name": [
        {
            "given": ["New"],
            "family": "Patient"
        }
    ],
    "birthDate": "2020-10-16"
}

And we update the last name to 'Provider'. The above version of the patient will be kept in _history, while in the 'Patient' store in the db, we will have the updated version:

{
    "resourceType": "Patient",
    "id": "fhirfli-294057507-6811107",
    "meta": {
        "versionId": "2",
        "lastUpdated": "2020-10-16T19:45:07.316698Z"
    },
    "name": [
        {
            "given": ["New"],
            "family": "Provider"
        }
    ],
    "birthDate": "2020-10-16"
}

This way we can keep track of all previous version of all resources (which is obviously important in medicine).

For most of the interactions (saving, deleting, etc), they work the way you'd expect. The only difference is search. Because Sembast is NoSQL, we can search on any of the fields in a resource. If in our interface class, we have the following function:

  Future<Either<DbFailure, List<Resource>>> searchFunction(
      String resourceType, String searchString, String reference) async {
    List<Resource> resultList;
    try {
      resultList =
          await resourceDao.searchFor(resourceType, searchString, reference);
    } catch (error) {
      return left(DbFailure.unableToObtainList(error: error.toString()));
    }
    return right(resultList);
  }

You can search for all immunizations of a certain patient:

searchFunction(
        'Immunization', 'patient.reference', 'Patient/$patientId');

This function will search through all entries in the 'Immunization' store. It will look at all 'patient.reference' fields, and return any that match 'Patient/$patientId'.

The last thing I'll mention is that this is a password protected db, using AES-256 encryption (although it can also use Salsa20). Anytime you use the db, you have the option of using a password for encryption/decryption. Remember, if you setup the database using encryption, you will only be able to access it using that same password. When you're ready to change the password, you will need to call the update password function. If we again assume we created a change password method in our interface, it might look something like this:

class IFhirDb {
  IFhirDb();
  final ResourceDao resourceDao = ResourceDao();
  ...
    Future<Either<DbFailure, Unit>> updatePassword(String oldPassword, String newPassword) async {
    try {
      await resourceDao.updatePw(oldPassword, newPassword);
    } catch (error) {
      return left(DbFailure.unableToUpdatePassword(error: error.toString()));
    }
    return right(Unit);
  }

You don't have to use a password, and in that case, it will save the db file as plain text. If you want to add a password later, it will encrypt it at that time.

General Store

After using this for a while in an app, I've realized that it needs to be able to store data apart from just FHIR resources, at least on occasion. For this, I've added a second class for all versions of the database called GeneralDao. This is similar to the ResourceDao, but fewer options. So, in order to save something, it would look like this:

await GeneralDao().save('password', {'new':'map'});
await GeneralDao().save('password', {'new':'map'}, 'key');

The difference between these two options is that the first one will generate a key for the map being stored, while the second will store the map using the key provided. Both will return the key after successfully storing the map.

Other functions available include:

// deletes everything in the general store
await GeneralDao().deleteAllGeneral('password'); 

// delete specific entry
await GeneralDao().delete('password','key'); 

// returns map with that key
await GeneralDao().find('password', 'key'); 

FHIR® is a registered trademark of Health Level Seven International (HL7) and its use does not constitute an endorsement of products by HL7®

Use this package as a library

Depend on it

Run this command:

With Flutter:

 $ flutter pub add fhir_db

This will add a line like this to your package's pubspec.yaml (and run an implicit flutter pub get):

dependencies:
  fhir_db: ^0.4.3

Alternatively, your editor might support or flutter pub get. Check the docs for your editor to learn more.

Import it

Now in your Dart code, you can use:

import 'package:fhir_db/dstu2.dart';
import 'package:fhir_db/dstu2/fhir_db.dart';
import 'package:fhir_db/dstu2/general_dao.dart';
import 'package:fhir_db/dstu2/resource_dao.dart';
import 'package:fhir_db/encrypt/aes.dart';
import 'package:fhir_db/encrypt/salsa.dart';
import 'package:fhir_db/r4.dart';
import 'package:fhir_db/r4/fhir_db.dart';
import 'package:fhir_db/r4/general_dao.dart';
import 'package:fhir_db/r4/resource_dao.dart';
import 'package:fhir_db/r5.dart';
import 'package:fhir_db/r5/fhir_db.dart';
import 'package:fhir_db/r5/general_dao.dart';
import 'package:fhir_db/r5/resource_dao.dart';
import 'package:fhir_db/stu3.dart';
import 'package:fhir_db/stu3/fhir_db.dart';
import 'package:fhir_db/stu3/general_dao.dart';
import 'package:fhir_db/stu3/resource_dao.dart'; 

example/lib/main.dart

import 'package:fhir/r4.dart';
import 'package:fhir_db/r4.dart';
import 'package:flutter/material.dart';
import 'package:test/test.dart';

Future<void> main() async {
  WidgetsFlutterBinding.ensureInitialized();

  final resourceDao = ResourceDao();

  // await resourceDao.updatePw('newPw', null);
  await resourceDao.deleteAllResources(null);

  group('Playing with passwords', () {
    test('Playing with Passwords', () async {
      final patient = Patient(id: Id('1'));

      final saved = await resourceDao.save(null, patient);

      await resourceDao.updatePw(null, 'newPw');
      final search1 = await resourceDao.find('newPw',
          resourceType: R4ResourceType.Patient, id: Id('1'));
      expect(saved, search1[0]);

      await resourceDao.updatePw('newPw', 'newerPw');
      final search2 = await resourceDao.find('newerPw',
          resourceType: R4ResourceType.Patient, id: Id('1'));
      expect(saved, search2[0]);

      await resourceDao.updatePw('newerPw', null);
      final search3 = await resourceDao.find(null,
          resourceType: R4ResourceType.Patient, id: Id('1'));
      expect(saved, search3[0]);

      await resourceDao.deleteAllResources(null);
    });
  });

  final id = Id('12345');
  group('Saving Things:', () {
    test('Save Patient', () async {
      final humanName = HumanName(family: 'Atreides', given: ['Duke']);
      final patient = Patient(id: id, name: [humanName]);
      final saved = await resourceDao.save(null, patient);

      expect(saved.id, id);

      expect((saved as Patient).name?[0], humanName);
    });

    test('Save Organization', () async {
      final organization = Organization(id: id, name: 'FhirFli');
      final saved = await resourceDao.save(null, organization);

      expect(saved.id, id);

      expect((saved as Organization).name, 'FhirFli');
    });

    test('Save Observation1', () async {
      final observation1 = Observation(
        id: Id('obs1'),
        code: CodeableConcept(text: 'Observation #1'),
        effectiveDateTime: FhirDateTime(DateTime(1981, 09, 18)),
      );
      final saved = await resourceDao.save(null, observation1);

      expect(saved.id, Id('obs1'));

      expect((saved as Observation).code.text, 'Observation #1');
    });

    test('Save Observation1 Again', () async {
      final observation1 = Observation(
          id: Id('obs1'),
          code: CodeableConcept(text: 'Observation #1 - Updated'));
      final saved = await resourceDao.save(null, observation1);

      expect(saved.id, Id('obs1'));

      expect((saved as Observation).code.text, 'Observation #1 - Updated');

      expect(saved.meta?.versionId, Id('2'));
    });

    test('Save Observation2', () async {
      final observation2 = Observation(
        id: Id('obs2'),
        code: CodeableConcept(text: 'Observation #2'),
        effectiveDateTime: FhirDateTime(DateTime(1981, 09, 18)),
      );
      final saved = await resourceDao.save(null, observation2);

      expect(saved.id, Id('obs2'));

      expect((saved as Observation).code.text, 'Observation #2');
    });

    test('Save Observation3', () async {
      final observation3 = Observation(
        id: Id('obs3'),
        code: CodeableConcept(text: 'Observation #3'),
        effectiveDateTime: FhirDateTime(DateTime(1981, 09, 18)),
      );
      final saved = await resourceDao.save(null, observation3);

      expect(saved.id, Id('obs3'));

      expect((saved as Observation).code.text, 'Observation #3');
    });
  });

  group('Finding Things:', () {
    test('Find 1st Patient', () async {
      final search = await resourceDao.find(null,
          resourceType: R4ResourceType.Patient, id: id);
      final humanName = HumanName(family: 'Atreides', given: ['Duke']);

      expect(search.length, 1);

      expect((search[0] as Patient).name?[0], humanName);
    });

    test('Find 3rd Observation', () async {
      final search = await resourceDao.find(null,
          resourceType: R4ResourceType.Observation, id: Id('obs3'));

      expect(search.length, 1);

      expect(search[0].id, Id('obs3'));

      expect((search[0] as Observation).code.text, 'Observation #3');
    });

    test('Find All Observations', () async {
      final search = await resourceDao.getResourceType(
        null,
        resourceTypes: [R4ResourceType.Observation],
      );

      expect(search.length, 3);

      final idList = [];
      for (final obs in search) {
        idList.add(obs.id.toString());
      }

      expect(idList.contains('obs1'), true);

      expect(idList.contains('obs2'), true);

      expect(idList.contains('obs3'), true);
    });

    test('Find All (non-historical) Resources', () async {
      final search = await resourceDao.getAll(null);

      expect(search.length, 5);
      final patList = search.toList();
      final orgList = search.toList();
      final obsList = search.toList();
      patList.retainWhere(
          (resource) => resource.resourceType == R4ResourceType.Patient);
      orgList.retainWhere(
          (resource) => resource.resourceType == R4ResourceType.Organization);
      obsList.retainWhere(
          (resource) => resource.resourceType == R4ResourceType.Observation);

      expect(patList.length, 1);

      expect(orgList.length, 1);

      expect(obsList.length, 3);
    });
  });

  group('Deleting Things:', () {
    test('Delete 2nd Observation', () async {
      await resourceDao.delete(
          null, null, R4ResourceType.Observation, Id('obs2'), null, null);

      final search = await resourceDao.getResourceType(
        null,
        resourceTypes: [R4ResourceType.Observation],
      );

      expect(search.length, 2);

      final idList = [];
      for (final obs in search) {
        idList.add(obs.id.toString());
      }

      expect(idList.contains('obs1'), true);

      expect(idList.contains('obs2'), false);

      expect(idList.contains('obs3'), true);
    });

    test('Delete All Observations', () async {
      await resourceDao.deleteSingleType(null,
          resourceType: R4ResourceType.Observation);

      final search = await resourceDao.getAll(null);

      expect(search.length, 2);

      final patList = search.toList();
      final orgList = search.toList();
      patList.retainWhere(
          (resource) => resource.resourceType == R4ResourceType.Patient);
      orgList.retainWhere(
          (resource) => resource.resourceType == R4ResourceType.Organization);

      expect(patList.length, 1);

      expect(patList.length, 1);
    });

    test('Delete All Resources', () async {
      await resourceDao.deleteAllResources(null);

      final search = await resourceDao.getAll(null);

      expect(search.length, 0);
    });
  });

  group('Password - Saving Things:', () {
    test('Save Patient', () async {
      await resourceDao.updatePw(null, 'newPw');
      final humanName = HumanName(family: 'Atreides', given: ['Duke']);
      final patient = Patient(id: id, name: [humanName]);
      final saved = await resourceDao.save('newPw', patient);

      expect(saved.id, id);

      expect((saved as Patient).name?[0], humanName);
    });

    test('Save Organization', () async {
      final organization = Organization(id: id, name: 'FhirFli');
      final saved = await resourceDao.save('newPw', organization);

      expect(saved.id, id);

      expect((saved as Organization).name, 'FhirFli');
    });

    test('Save Observation1', () async {
      final observation1 = Observation(
        id: Id('obs1'),
        code: CodeableConcept(text: 'Observation #1'),
        effectiveDateTime: FhirDateTime(DateTime(1981, 09, 18)),
      );
      final saved = await resourceDao.save('newPw', observation1);

      expect(saved.id, Id('obs1'));

      expect((saved as Observation).code.text, 'Observation #1');
    });

    test('Save Observation1 Again', () async {
      final observation1 = Observation(
          id: Id('obs1'),
          code: CodeableConcept(text: 'Observation #1 - Updated'));
      final saved = await resourceDao.save('newPw', observation1);

      expect(saved.id, Id('obs1'));

      expect((saved as Observation).code.text, 'Observation #1 - Updated');

      expect(saved.meta?.versionId, Id('2'));
    });

    test('Save Observation2', () async {
      final observation2 = Observation(
        id: Id('obs2'),
        code: CodeableConcept(text: 'Observation #2'),
        effectiveDateTime: FhirDateTime(DateTime(1981, 09, 18)),
      );
      final saved = await resourceDao.save('newPw', observation2);

      expect(saved.id, Id('obs2'));

      expect((saved as Observation).code.text, 'Observation #2');
    });

    test('Save Observation3', () async {
      final observation3 = Observation(
        id: Id('obs3'),
        code: CodeableConcept(text: 'Observation #3'),
        effectiveDateTime: FhirDateTime(DateTime(1981, 09, 18)),
      );
      final saved = await resourceDao.save('newPw', observation3);

      expect(saved.id, Id('obs3'));

      expect((saved as Observation).code.text, 'Observation #3');
    });
  });

  group('Password - Finding Things:', () {
    test('Find 1st Patient', () async {
      final search = await resourceDao.find('newPw',
          resourceType: R4ResourceType.Patient, id: id);
      final humanName = HumanName(family: 'Atreides', given: ['Duke']);

      expect(search.length, 1);

      expect((search[0] as Patient).name?[0], humanName);
    });

    test('Find 3rd Observation', () async {
      final search = await resourceDao.find('newPw',
          resourceType: R4ResourceType.Observation, id: Id('obs3'));

      expect(search.length, 1);

      expect(search[0].id, Id('obs3'));

      expect((search[0] as Observation).code.text, 'Observation #3');
    });

    test('Find All Observations', () async {
      final search = await resourceDao.getResourceType(
        'newPw',
        resourceTypes: [R4ResourceType.Observation],
      );

      expect(search.length, 3);

      final idList = [];
      for (final obs in search) {
        idList.add(obs.id.toString());
      }

      expect(idList.contains('obs1'), true);

      expect(idList.contains('obs2'), true);

      expect(idList.contains('obs3'), true);
    });

    test('Find All (non-historical) Resources', () async {
      final search = await resourceDao.getAll('newPw');

      expect(search.length, 5);
      final patList = search.toList();
      final orgList = search.toList();
      final obsList = search.toList();
      patList.retainWhere(
          (resource) => resource.resourceType == R4ResourceType.Patient);
      orgList.retainWhere(
          (resource) => resource.resourceType == R4ResourceType.Organization);
      obsList.retainWhere(
          (resource) => resource.resourceType == R4ResourceType.Observation);

      expect(patList.length, 1);

      expect(orgList.length, 1);

      expect(obsList.length, 3);
    });
  });

  group('Password - Deleting Things:', () {
    test('Delete 2nd Observation', () async {
      await resourceDao.delete(
          'newPw', null, R4ResourceType.Observation, Id('obs2'), null, null);

      final search = await resourceDao.getResourceType(
        'newPw',
        resourceTypes: [R4ResourceType.Observation],
      );

      expect(search.length, 2);

      final idList = [];
      for (final obs in search) {
        idList.add(obs.id.toString());
      }

      expect(idList.contains('obs1'), true);

      expect(idList.contains('obs2'), false);

      expect(idList.contains('obs3'), true);
    });

    test('Delete All Observations', () async {
      await resourceDao.deleteSingleType('newPw',
          resourceType: R4ResourceType.Observation);

      final search = await resourceDao.getAll('newPw');

      expect(search.length, 2);

      final patList = search.toList();
      final orgList = search.toList();
      patList.retainWhere(
          (resource) => resource.resourceType == R4ResourceType.Patient);
      orgList.retainWhere(
          (resource) => resource.resourceType == R4ResourceType.Organization);

      expect(patList.length, 1);

      expect(patList.length, 1);
    });

    test('Delete All Resources', () async {
      await resourceDao.deleteAllResources('newPw');

      final search = await resourceDao.getAll('newPw');

      expect(search.length, 0);

      await resourceDao.updatePw('newPw', null);
    });
  });
} 

Download Details:

Author: MayJuun

Source Code: https://github.com/MayJuun/fhir/tree/main/fhir_db

#sqflite  #dart  #flutter 

伊藤  直子

伊藤 直子

1633693200

【 初心者向け】C言語でのマルチスレッド の概要

ニューヨークで働き、ウォール街中のプログラマーと話をしていると、ほとんどのリアルタイムプログラミングアプリケーションで期待される共通の知識の糸に気づきました。その知識はマルチスレッドとして知られています。私はプログラミングの世界を移動し、潜在的なプログラミング候補者にインタビューを行ったので、マルチスレッドについてほとんど知られていないことや、スレッドが適用される理由や方法に驚かされることは決してありません。Vance Morrisonによって書かれた一連の優れた記事で、MSDNはこの問題に対処しようとしました:(MSDNの8月号、すべての開発者がマルチスレッドアプリについて知っておくべきこと、および10月号はマルチスレッドでのローロック技術の影響を理解するを参照してください)。アプリ

この記事では、スレッド化、スレッド化が使用される理由、および.NETでのスレッド化の使用方法について紹介します。マルチスレッドの背後にある謎を完全に明らかにし、それを説明する際に、コード内の潜在的なスレッド障害を回避するのに役立つことを願っています。

スレッドとは何ですか?

すべてのアプリケーションは、少なくとも1つのスレッドで実行されます。では、スレッドとは何ですか?スレッドはプロセスにすぎません。私の推測では、糸という言葉は、織機で糸を織り上げる超自然的なミューズのギリシャ神話に由来していると思います。各糸は、誰かの人生の時間の道を表しています。あなたがその糸をいじると、あなたは人生の構造を乱したり、人生のプロセスを変えたりします。コンピューターでは、スレッドは時間の経過とともに移動するプロセスです。プロセスは一連の順次ステップを実行し、各ステップはコード行を実行します。ステップは連続しているため、各ステップには一定の時間がかかります。一連のステップを完了するのにかかる時間は、各プログラミングステップの実行にかかる時間の合計です。

マルチスレッドアプリケーションとは何ですか?

長い間、ほとんどのプログラミングアプリケーション(組み込みシステムプログラムを除く)はシングルスレッドでした。これは、アプリケーション全体でスレッドが1つしかないことを意味します。計算Bが完了するまで、計算Aを実行することはできません。プログラムはステップ1から始まり、最後のステップ(ステップ10と呼びます)に到達するまで順次続行します(ステップ2、ステップ3、ステップ4)。マルチスレッドアプリケーションを使用すると、複数のスレッドを実行できます。各スレッドは独自のプロセスで実行されます。したがって、理論的には、あるスレッドでステップ1を実行し、同時に別のスレッドでステップ2を実行できます。同時に、ステップ3を独自のスレッドで実行し、ステップ4を独自のスレッドで実行することもできます。したがって、ステップ1、ステップ2、ステップ3、およびステップ4は同時に実行されます。理論的には、4つのステップすべてがほぼ同時にかかった場合、シングルスレッドの実行にかかる時間の4分の1でプログラムを終了できます(4プロセッサマシンを使用していると仮定)。では、なぜすべてのプログラムがマルチスレッド化されていないのでしょうか。スピードとともに、あなたは複雑さに直面するからです。ステップ1がステップ2の情報に何らかの形で依存している場合を想像してみてください。ステップ1がステップ2の前に計算を終了した場合、またはその逆の場合、プログラムが正しく実行されない可能性があります。

珍しいアナロジー

マルチスレッドを考える別の方法は、人体を考慮することです。体の各器官(心臓、肺、肝臓、脳)はすべてプロセスに関与しています。各プロセスは同時に実行されています。各臓器がプロセスのステップとして実行された場合を想像してみてください。最初に心臓、次に脳、次に肝臓、次に肺です。私たちはおそらく死んでしまうでしょう。つまり、人体は1つの大きなマルチスレッドアプリケーションのようなものです。すべての臓器は同時に実行されるプロセスであり、これらのプロセスはすべて相互に依存しています。これらのプロセスはすべて、神経信号、血流、化学的トリガーを介して通信します。すべてのマルチスレッドアプリケーションと同様に、人体は非常に複雑です。一部のプロセスが他のプロセスから情報を取得しない場合、または特定のプロセスが遅くなったり速くなったりすると、医学的な問題が発生します。それか'

いつスレッド化するか

マルチスレッドは、プログラムをより効率的に実行したい状況で最もよく使用されます。たとえば、ウィンドウフォームプログラムの中に、実行に1秒以上かかり、繰り返し実行する必要のあるメソッド(method_Aと呼びます)が含まれているとします。プログラム全体が単一のスレッドで実行された場合、ボタンの押下が正しく機能しなかったり、入力が少し遅くなったりすることがあります。method_Aの計算量が多すぎると、ウィンドウフォームの特定の部分がまったく機能しないことに気付くかもしれません。この許容できないプログラムの動作は、プログラムにマルチスレッドが必要であることを示しています。スレッド化が必要になるもう1つの一般的なシナリオは、メッセージングシステムです。アプリケーションに多数のメッセージが送信されている場合は、メインの処理プログラムの実行と同時にそれらをキャプチャし、適切に配布する必要があります。重い処理を実行しているときに一連のメッセージを効率的にキャプチャすることはできません。そうしないと、メッセージを見逃す可能性があります。複数のスレッドは、複数のプロセスが同時に実行される組立ライン方式で使用することもできます。たとえば、プロセスがスレッドでデータを収集すると、1つのプロセスがデータをフィルタリングし、1つのプロセスがデータをデータベースと照合します。これらの各シナリオはマルチスレッドの一般的な使用法であり、シングルスレッドで実行されている同様のアプリケーションのパフォーマンスを大幅に向上させます。そうしないと、メッセージを見逃す可能性があるためです。複数のスレッドは、複数のプロセスが同時に実行される組立ライン方式で使用することもできます。たとえば、プロセスがスレッドでデータを収集すると、1つのプロセスがデータをフィルタリングし、1つのプロセスがデータをデータベースと照合します。これらの各シナリオはマルチスレッドの一般的な使用法であり、シングルスレッドで実行されている同様のアプリケーションのパフォーマンスを大幅に向上させます。そうしないと、メッセージを見逃す可能性があるためです。複数のスレッドは、複数のプロセスが同時に実行される組立ライン方式で使用することもできます。たとえば、プロセスがスレッドでデータを収集すると、1つのプロセスがデータをフィルタリングし、1つのプロセスがデータをデータベースと照合します。これらの各シナリオはマルチスレッドの一般的な使用法であり、シングルスレッドで実行されている同様のアプリケーションのパフォーマンスを大幅に向上させます。

スレッドしない場合

初心者のプログラマーが最初にスレッド化を学ぶとき、彼らはプログラムでスレッド化を使用する可能性に魅了される可能性があります。彼らは実際にスレッドハッピーになるかもしれません  詳しく説明させてください、

1日目)プログラマーは、スレッドを生成できることを学び、プログラムで1つの新しいスレッドCool!の作成を開始します 。

2日目)プログラマーは、「プログラムの一部で他のスレッドを生成することで、これをさらに効率的にすることができます!」と言います。

3日目)P:「わあ、スレッド内でスレッドをフォークすることもでき、本当に効率が向上します!!」

4日目)P:「奇妙な結果が出ているようですが、それは問題ありません。今は無視します。」

5日目)「うーん、widgetX変数に値がある場合もありますが、まったく設定されていないように見える場合もあります。コンピューターが機能していないため、デバッガーを実行するだけです」。

9日目)「このくそったれ(より強い言語)プログラムはあちこちでジャンプしています!!何が起こっているのか理解できません!」

2週目)時々、プログラムはただそこに座って、まったく何もしません!ヘルプ!!!!!

おなじみですか?マルチスレッドプログラムを初めて設計しようとしたほとんどの人は、スレッドの設計知識が豊富であっても、おそらくこれらの毎日の箇条書きの少なくとも1つまたは2つを経験したことがあります。スレッド化が悪いことだとほのめかしているわけではありません。プログラムでスレッド化の効率を上げるプロセスでは、非常に注意してください。  シングルスレッドプログラムとは異なり、同時に多くのプロセスを処理しているため、複数の従属変数を持つ複数のプロセスを追跡するのは非常に難しい場合があります。ジャグリングと同じようにマルチスレッドを考えてください。手で1つのボールをジャグリングするのは(退屈ではありますが)かなり簡単です。ただし、これらのボールのうち2つを空中に置くように挑戦された場合、その作業は少し難しくなります。3、4、および5の場合、ボールは次第に難しくなります。ボールの数が増えると、実際にボールを落とす可能性が高くなります。 一度にたくさんのボールをジャグリングするには、知識、スキル、正確なタイミングが必要です。マルチスレッドもそうです。 

マルチスレッド

図1-マルチスレッドはジャグリングプロセスのようなものです

 
スレッディングの問題

プログラム内のすべてのプロセスが相互に排他的である場合、つまり、プロセスが他のプロセスにまったく依存していない場合、複数のスレッド化は非常に簡単で、問題はほとんど発生しません。各プロセスは、他のプロセスに煩わされることなく、独自のハッピーコースで実行されます。ただし、複数のプロセスが他のプロセスによって使用されているメモリの読み取りまたは書き込みを行う必要がある場合、問題が発生する可能性があります。たとえば、プロセス#1とプロセス#2の2つのプロセスがあるとします。両方のプロセスが変数Xを共有します。スレッドプロセス#1が最初に値5の変数Xを書き込み、スレッドプロセス#2が次に値-3の変数Xを書き込む場合、Xの最終値は-3です。ただし、プロセス#2が最初に値-3の変数Xを書き込み、次にプロセス#1が値5の変数Xを書き込む場合、Xの最終値は5です。Xを設定できるプロセスがプロセス#1またはプロセス#2の知識を持っていない場合、Xは、最初にXに到達したスレッドに応じて異なる最終値になる可能性があります。シングルスレッドプログラムでは、すべてが順番に続くため、これが発生する可能性はありません。シングルスレッドプログラムでは、並列に実行されているプロセスがないため、Xは常に最初にメソッド#1によって設定され(最初に呼び出された場合)、次にメソッド#2によって設定されます。シングルスレッドプログラムには驚きはありません。それはステップバイステップです。マルチスレッドプログラムを使用すると、2つのスレッドが同時にコードを入力し、結果に大混乱をもたらす可能性があります。スレッドの問題は、同時に実行されている別のスレッドが同じコードを入力して共有データを操作できるようにしながら、共有メモリにアクセスする1つのスレッドを制御する何らかの方法が必要なことです。Xは、どのスレッドが最初にXに到達したかによって、最終的に異なる最終値になる可能性があります。シングルスレッドプログラムでは、すべてが順番に続くため、これが発生する可能性はありません。シングルスレッドプログラムでは、並列に実行されているプロセスがないため、Xは常に最初にメソッド#1によって設定され(最初に呼び出された場合)、次にメソッド#2によって設定されます。シングルスレッドプログラムには驚きはありません。それはステップバイステップです。マルチスレッドプログラムを使用すると、2つのスレッドが同時にコードを入力し、結果に大混乱をもたらす可能性があります。スレッドの問題は、同時に実行されている別のスレッドが同じコードを入力して共有データを操作できるようにしながら、共有メモリにアクセスする1つのスレッドを制御する何らかの方法が必要なことです。Xは、どのスレッドが最初にXに到達したかによって、最終的に異なる最終値になる可能性があります。シングルスレッドプログラムでは、すべてが順番に続くため、これが発生する可能性はありません。シングルスレッドプログラムでは、並列に実行されているプロセスがないため、Xは常に最初にメソッド#1によって設定され(最初に呼び出された場合)、次にメソッド#2によって設定されます。シングルスレッドプログラムには驚きはありません。それはステップバイステップです。マルチスレッドプログラムを使用すると、2つのスレッドが同時にコードを入力し、結果に大混乱をもたらす可能性があります。スレッドの問題は、同時に実行されている別のスレッドが同じコードを入力して共有データを操作できるようにしながら、共有メモリにアクセスする1つのスレッドを制御する何らかの方法が必要なことです。すべてが順番に続くため、これが発生する可能性はありません。シングルスレッドプログラムでは、並列に実行されているプロセスがないため、Xは常に最初にメソッド#1によって設定され(最初に呼び出された場合)、次にメソッド#2によって設定されます。シングルスレッドプログラムには驚きはありません。それはステップバイステップです。マルチスレッドプログラムを使用すると、2つのスレッドが同時にコードを入力し、結果に大混乱をもたらす可能性があります。スレッドの問題は、同時に実行されている別のスレッドが同じコードを入力して共有データを操作できるようにしながら、共有メモリにアクセスする1つのスレッドを制御する何らかの方法が必要なことです。すべてが順番に続くため、これが発生する可能性はありません。シングルスレッドプログラムでは、並列に実行されているプロセスがないため、Xは常に最初にメソッド#1によって設定され(最初に呼び出された場合)、次にメソッド#2によって設定されます。シングルスレッドプログラムには驚きはありません。それはステップバイステップです。マルチスレッドプログラムを使用すると、2つのスレッドが同時にコードを入力し、結果に大混乱をもたらす可能性があります。スレッドの問題は、同時に実行されている別のスレッドが同じコードを入力して共有データを操作できるようにしながら、共有メモリにアクセスする1つのスレッドを制御する何らかの方法が必要なことです。(最初に呼び出された場合)次に、メソッド#2で設定します。シングルスレッドプログラムには驚きはありません。それはステップバイステップです。マルチスレッドプログラムを使用すると、2つのスレッドが同時にコードを入力し、結果に大混乱をもたらす可能性があります。スレッドの問題は、同時に実行されている別のスレッドが同じコードを入力して共有データを操作できるようにしながら、共有メモリにアクセスする1つのスレッドを制御する何らかの方法が必要なことです。(最初に呼び出された場合)次に、メソッド#2で設定します。シングルスレッドプログラムには驚きはありません。それはステップバイステップです。マルチスレッドプログラムを使用すると、2つのスレッドが同時にコードを入力し、結果に大混乱をもたらす可能性があります。スレッドの問題は、同時に実行されている別のスレッドが同じコードを入力して共有データを操作できるようにしながら、共有メモリにアクセスする1つのスレッドを制御する何らかの方法が必要なことです。 

スレッドセーフ

3つのボールをジャグリングするたびに、空中のボールが、自然の異常によって、すでに右手に座っているボールが投げられるまで、右手に到達することが決して許されなかったと想像してみてください。少年、ジャグリングはずっと簡単でしょう!これがスレッドセーフのすべてです。私たちのプログラムでは、もう一方のスレッドがビジネスを終了している間、一方のスレッドをコードブロック内で待機させます。スレッドのブロックまたはスレッドの同期と呼ばれるこのアクティビティにより、プログラム内で実行される同時スレッドのタイミングを制御できます。C#では、メモリの特定の部分(通常はオブジェクトのインスタンス)をロックし、オブジェクトを使用して別のスレッドが完了するまで、スレッドがこのオブジェクトのメモリのコードを入力できないようにします。今ではおそらくコード例を渇望しているので、ここに行きます。

2スレッドのシナリオを見てみましょう。この例では、C#でスレッド1とスレッド2の2つのスレッドを作成します。どちらも、独自のwhileループで実行されます。スレッドは何の役にも立ちません。どのスレッドに属しているかを示すメッセージを出力するだけです。_threadOutputと呼ばれる共有メモリクラスメンバーを利用します。_threadOutputには、実行中のスレッドに基づいてメッセージが割り当てられます。リスト#1は、それぞれDisplayThread1とDisplayThread2に含まれる2つのスレッドを示しています。

リスト1-メモリ内で共通の変数を共有する2つのスレッドを作成する

// shared memory variable between the two threads  
// used to indicate which thread we are in  
private string _threadOutput = "";  
  
/// <summary>  
/// Thread 1: Loop continuously,  
/// Thread 1: Displays that we are in thread 1  
/// </summary>  
void DisplayThread1()  
{  
      while (_stopThreads == false)  
      {  
            Console.WriteLine("Display Thread 1");  
  
            // Assign the shared memory to a message about thread #1  
            _threadOutput = "Hello Thread1";  
  
  
            Thread.Sleep(1000);  // simulate a lot of processing   
  
            // tell the user what thread we are in thread #1, and display shared memory  
            Console.WriteLine("Thread 1 Output --> {0}", _threadOutput);  
  
      }  
}  

/// <summary>  
/// Thread 2: Loop continuously,  
/// Thread 2: Displays that we are in thread 2  
/// </summary>  
void DisplayThread2()  
{  
      while (_stopThreads == false)  
      {  
        Console.WriteLine("Display Thread 2");  
  
  
       // Assign the shared memory to a message about thread #2  
        _threadOutput = "Hello Thread2";  
  
  
        Thread.Sleep(1000);  // simulate a lot of processing  
  
       // tell the user we are in thread #2  
        Console.WriteLine("Thread 2 Output --> {0}", _threadOutput);  
  
      }  
}
Class1()  
{  
      // construct two threads for our demonstration;  
      Thread thread1 = new Thread(new ThreadStart(DisplayThread1));  
      Thread thread2 = new Thread(new ThreadStart(DisplayThread2));  
  
      // start them  
      thread1.Start();  
      thread2.Start();  
}

このコードの結果を図2に示します。結果を注意深く見てください。プログラムが驚くべき出力を提供することに気付くでしょう(これをシングルスレッドの考え方から見た場合)。_threadOutputを、それが属するスレッドに対応する番号の文字列に明確に割り当てましたが、コンソールでは次のように表示されます。

C#でのスレッド化

図2-2スレッドの例からの異常な出力。

私たちのコードから次のことが期待されます、

スレッド1の出力->ハロースレッド1とスレッド2の出力->ハロースレッド2ですが、ほとんどの場合、結果は完全に予測できません。 

スレッド2の出力->ハロースレッド1とスレッド1の出力->ハロースレッド2が表示されることがあります。スレッドの出力がコードと一致しません。コードを見て、それを目で追っていますが、_threadOutput = "Hello Thread 2"、Sleep、Write "Thread 2-> Hello Thread 2"ですが、このシーケンスで必ずしも最終結果が得られるとは限りません。 

説明

このようなマルチスレッドプログラムでは、理論的にはコードが2つのメソッドDisplayThread1とDisplayThread2を同時に実行しているためです。各メソッドは変数_threadOutputを共有します。したがって、_threadOutputにはスレッド#1で値 "Hello Thread1"が割り当てられ、2行後にコンソールに_threadOutputが表示されますが、スレッド#1がそれを割り当てて表示する時間の間のどこかで、スレッド#2が_threadOutputを割り当てる可能性があります。値「HelloThread2」。これらの奇妙な結果が発生する可能性があるだけでなく、図2に示す出力に見られるように、非常に頻繁に発生します。この痛みを伴うスレッドの問題は、競合状態として知られるスレッドプログラミングで非常に一般的なバグです。 この例は、よく知られているスレッドの問題の非常に単純な例です。この問題は、参照されている変数やスレッドセーフでない変数を指すコレクションなどを介して、プログラマーからはるかに間接的に隠されている可能性があります。図2では症状は露骨ですが、競合状態は非常にまれにしか現れず、1分に1回、1時間に1回、または3日後に断続的に現れる可能性があります。レースは、その頻度が低く、再現が非常に難しいため、おそらくプログラマーにとって最悪の悪夢です。

レースに勝つ

競合状態を回避する最善の方法は、スレッドセーフなコードを作成することです。コードがスレッドセーフである場合、いくつかの厄介なスレッドの問題が発生するのを防ぐことができます。スレッドセーフなコードを書くためのいくつかの防御策があります。1つは、メモリの共有をできるだけ少なくすることです。クラスのインスタンスを作成し、それが1つのスレッドで実行され、次に同じクラスの別のインスタンスを作成し、それが別のスレッドで実行される場合、静的変数が含まれていない限り、クラスはスレッドセーフです。 。2つのクラスはそれぞれ、独自のフィールド用に独自のメモリを作成するため、共有メモリはありません。クラスに静的変数がある場合、またはクラスのインスタンスが他の複数のスレッドによって共有されている場合は、他のクラスがその変数の使用を完了するまで、一方のスレッドがその変数のメモリを使用できないようにする方法を見つける必要があります。ロック。  C#を使用すると、Monitorクラスまたはlock {}構造のいずれかを使用してコードをロックできます。(lock構造は、実際にはtry-finallyブロックを介してMonitorクラスを内部的に実装しますが、プログラマーからこれらの詳細を隠します)。リスト1の例では、共有_threadOutput変数を設定した時点から、コンソールへの実際の出力まで、コードのセクションをロックできます。コードのクリティカルセクションを両方のスレッドでロックして、どちらか一方に競合が発生しないようにします。メソッド内をロックする最も速くて汚い方法は、このポインターをロックすることです。このポインタをロックすると、クラスインスタンス全体がロックされるため、ロック内でクラスのフィールドを変更しようとするスレッドはすべてブロックされます。。ブロッキングとは、変数を変更しようとしているスレッドが、ロックされたスレッドでロックが解除されるまで待機することを意味します。スレッドは、lock {}構造の最後のブラケットに到達すると、ロックから解放されます。

リスト2-2つのスレッドをロックして同期する

/// <summary>  
/// Thread 1, Displays that we are in thread 1 (locked)  
 /// </summary>  
 void DisplayThread1()  
 {  
       while (_stopThreads == false)  
       {  
          // lock on the current instance of the class for thread #1  
             lock (this)  
             {  
                   Console.WriteLine("Display Thread 1");  
                   _threadOutput = "Hello Thread1";  
                   Thread.Sleep(1000);  // simulate a lot of processing  
                   // tell the user what thread we are in thread #1  
                   Console.WriteLine("Thread 1 Output --> {0}", _threadOutput);  
             }// lock released  for thread #1 here  
       }   
 }  

/// <summary>  
/// Thread 1, Displays that we are in thread 1 (locked)  
 /// </summary>  
 void DisplayThread2()  
 {  
       while (_stopThreads == false)  
       {  
  
           // lock on the current instance of the class for thread #2  
             lock (this)  
             {  
                   Console.WriteLine("Display Thread 2");  
                   _threadOutput = "Hello Thread2";  
                   Thread.Sleep(1000);  // simulate a lot of processing  
                   // tell the user what thread we are in thread #1  
                   Console.WriteLine("Thread 2 Output --> {0}", _threadOutput);  
             } // lock released  for thread #2 here  
       }   
 }

2つのスレッドをロックした結果を図3に示します。すべてのスレッド出力が適切に同期されていることに注意してください。スレッド1の出力->ハロースレッド1とスレッド2の出力->ハロースレッド2という結果が常に表示されます。ただし、スレッドのロックにはコストがかかることに注意してください。スレッドをロックすると、ロックが解除されるまで他のスレッドを強制的に待機させます。本質的に、他のスレッドが共有メモリの使用を待機している間、最初のスレッドはプログラムで何もしていないため、プログラムの速度が低下しました。したがって、ロックは慎重に使用する必要があります。共有メモリに関与していない場合は、コード内にあるすべてのメソッドをロックするだけではいけません。また、ロックを使用するときは注意してください。スレッド#1がスレッド#2によってロックが解放されるのを待っている状況に陥りたくないからです。スレッド#2は、スレッド#1によってロックが解放されるのを待っています。この状況が発生すると、両方のスレッドがブロックされ、プログラムがフリーズしたように見えます。この状況はとして知られていますデッドロックが発生し、プログラム内の予測できない断続的な期間にも発生する可能性があるため、競合状態とほぼ同じくらい悪い状況です。 

  C#でのスレッド化

  図3-ロックを使用したデュアルスレッドプログラムの同期

代替ソリューション

.NETは、スレッドの制御に役立つ多くのメカニズムを提供します。別のスレッドが共有メモリの一部を処理している間、スレッドをブロックしたままにする別の方法は、AutoResetEventを使用することです。AutoResetEventクラスには、SetとWaitOneの2つのメソッドがあります。これらの2つの方法は、スレッドのブロックを制御するために一緒に使用できます。AutoResetEventがfalseで初期化されると、プログラムは、AutoResetEventでSetメソッドが呼び出されるまで、WaitOneを呼び出すコード行で停止します。AutoResetEventでSetメソッドが実行されると、スレッドのブロックが解除され、WaitOneを超えて続行できるようになります。次回WaitOneが呼び出されると、自動的にリセットされるため、プログラムは、WaitOneメソッドが実行されているコード行で再び待機(ブロック)します。この「停止とトリガー」を使用できます Setを呼び出して、別のスレッドがブロックされたスレッドを解放する準備ができるまで、あるスレッドをブロックするメカニズム。リスト3は、AutoResetEventを使用して、ブロックされたスレッドが待機し、ブロックされていないスレッドが実行されてコンソールに_threadOutputを表示している間に、互いにブロックする同じ2つのスレッドを示しています。最初に、_blockThread1はfalseを通知するように初期化され、_blockThread2はtrueを通知するように初期化されます。これは、_blockThread2がDisplayThread_2のループを最初に通過するときに、WaitOne呼び出しを続行できるようになる一方で、_blockThread1はDisplayThread_1のWaitOne呼び出しをブロックすることを意味します。_blockThread2がスレッド2のループの終わりに達すると、スレッド1をブロックから解放するためにSetを呼び出して_blockThread1に信号を送ります。次に、スレッド2は、スレッド1がループの終わりに到達して_blockThread2でSetを呼び出すまで、WaitOne呼び出しで待機します。スレッド1で呼び出されたセットはスレッド2のブロックを解放し、プロセスが再開されます。両方のAutoResetEvents(_blockThread1と_blockThread2)を最初にfalseを通知するように設定した場合、両方のスレッドが互いにトリガーする機会なしにループの進行を待機し、デッドロック。 

リスト3-あるいは、AutoResetEventでスレッドをブロックする

AutoResetEvent _blockThread1 = new AutoResetEvent(false);  
AutoResetEvent _blockThread2 = new AutoResetEvent(true);  
  
/// <summary>  
/// Thread 1, Displays that we are in thread 1  
/// </summary>  
void DisplayThread_1()  
{  
      while (_stopThreads == false)  
      {  
               // block thread 1  while the thread 2 is executing  
                _blockThread1.WaitOne();   
  
                // Set was called to free the block on thread 1, continue executing the code  
                  Console.WriteLine("Display Thread 1");  
  
                  _threadOutput = "Hello Thread 1";  
                  Thread.Sleep(1000);  // simulate a lot of processing  
  
                   // tell the user what thread we are in thread #1  
                  Console.WriteLine("Thread 1 Output --> {0}", _threadOutput);  
  
                // finished executing the code in thread 1, so unblock thread 2  
                  _blockThread2.Set();  
      }  
}  
  
/// <summary>  
/// Thread 2, Displays that we are in thread 2  
/// </summary>  
void DisplayThread_2()  
{  
      while (_stopThreads == false)  
      {  
            // block thread 2  while thread 1 is executing  
                  _blockThread2.WaitOne();   
  
            // Set was called to free the block on thread 2, continue executing the code  
                  Console.WriteLine("Display Thread 2");  
  
                  _threadOutput = "Hello Thread 2";  
                  Thread.Sleep(1000);  // simulate a lot of processing  
  
                   // tell the user we are in thread #2  
                  Console.WriteLine("Thread 2 Output --> {0}", _threadOutput);   
  
            // finished executing the code in thread 2, so unblock thread 1  
                _blockThread1.Set();  
      }  
} 

 

リスト3で生成される出力は、図3に示すロックコードと同じ出力ですが、AutoResetEventを使用すると、現在のスレッドが処理を完了したときに、あるスレッドが別のスレッドに通知する方法をより動的に制御できます。

結論

マイクロプロセッサの速度の理論的限界を押し上げているため、テクノロジは、コンピュータテクノロジの速度とパフォーマンスを最適化できる新しい方法を見つける必要があります。マルチプロセッサチップの発明と並列プログラミングへの侵入により、マルチスレッドを理解することで、ムーアの法則に挑戦し続けるために必要な利点をもたらすこれらのより最近のテクノロジーを処理するために必要なパラダイムに備えることができます。C#と.NETは、マルチスレッドと並列処理をサポートする機能を提供します。これらのツールを上手に活用する方法を理解すれば、私たち自身の日々のプログラミング活動において、将来のこれらのハードウェアの約束に備えることができます。一方、シャープなあなたができるので、スレッドのあなたの知識エン.NET可能性を。 

リンク:https://www.c-sharpcorner.com/article/introduction-to-multithreading-in-C-Sharp/

#csharp