Open-source intelligence

Template:Wikify-date Open source intelligence or OSINT refers to an intelligence gathering discipline based on analyzing information collected from open sources, i.e. information available to the public, both general and localized (e.g. school papers, church bulletins, or company information that is not proprietary). OSINT comes from five primary sources:

• Broadcasts (e.g., radio and television);
• Documents (e.g., reports, papers, and studies from academia and government institutions); and the
• Internet (e.g., websites, forums, and blogs).
• Experts (e.g. those identified via citation analysis, or those unpublished epxerts identified by published experts)
• Citizens (e.g. direct observation or life experience relevant to the question at hand)

Term

The term OSINT is related to the term open source as it applies to the five "opens" (open source software, open source intelligence, open spectrum, open access copyright, and Open Society) but in this context, is distinct from the much better organized and much more reliable free/open source software (F/OSS) social network thst shares source code which is publicly available (and modifiable). OSINT should also not be generally confused with OSIF (Open Source Information) on which OSINT is based. OSIF is any information that is publicly available; OSINT is an analytically-tailored intelligence product composed of OSIF which is designed to answer a specific tasking or to support decision-making.

The U.S. IC in particular has an information technology architecture, as well as a management mind-set, a legal mind-set, and security mind-set, that demands that ALL information be stored (and generally not shared) at the Top Secret Codeword level.

Collection in OSINT is generally a different problem from collection in other intelligence disciplines where obtaining the raw information to be analyzed may be a major difficultly, particularly if it is to be obtained from non-cooperative targets. In OSINT, the chief difficulty is in identifying relevant, reliable sources from the vast amount of publicly available information. However, this is not as great a challenge for those who know how to access local knowledge and how to leverage human experts who can create new tailored knowledge on the fly.

Overt Human Intelligence (HUMINT) is the use of non-clandestine human information sources that are approved by security officers who have very limited educational and real-world backgrounds, for example, the interrogation of refugees, debriefing of legal travellers, and public reports from overt agents such as attachés and ambassadors. Truly professional OSINT practitioners understand that EVERY human being, including bus drivers, secretaries, priests, teachers, and so on, have something to contribute to historical, cultural, or contextual understanding. Similarly, the most effective intelligence managers understand that the best intelligence professionals will often have complex international backgrounds and will have spend decades dealing with foreigners--these managers are often limited to entry level employees who have no foreign experience or foreign contacts, simply because they are easier for the security officers to approve for employment. In a robust international OSINT and information sharing environment, such pathological approaches to employment and interaction are moot.

OSINT is now a subordinate part of modern Information Operations (IO), which consists of Strategic Communication (the message), OSINT (the reality), and Joint Intelligence or Inter-Agency Collaboration or Coordination Centers or Commands (JIOC). The U.S. military is heavily invested in OSINT, with the U.S. Strategic Command having the lead for IO, the U.S. Special Operations Command having the lead for the best operationally-oriented OSINT capability, and the U.S. Central Command being the most important consumer of tailored operationally-oriented OSINT.

Current NATO guides to OSINT are the NATO Open Source Intelligence Handbook, the NATO Open Source Intelligence Reader, and the NATO Intelligence Exploitation of the Internet guide.

A history of OSINT in recent times is contained in the 30 volumes (30,000 pages of Proceedings from the annual OSINT conference sponsored by OSS.Net since 1992. The next conference will be in January 2007.

Why Open Source Intelligence is Important to the Public and to National Security:

According to The Commission on the Intelligence Capabilities of the United States Regarding Weapons of Mass Destruction report submitted in March 2005, OSINT must be included in the all-source SECRET intelligence process for the following reasons (as stated in the report):

1. The ever-shifting nature of our intelligence needs compels the IC to quickly and easily understand a wide range of foreign countries and cultures. - … today’s threats are rapidly changing and geographically diffuse; it is a fact of life that an intelligence analyst may be forced to shift rapidly from one topic to the next. Increasingly, IC professionals need to quickly assimilate social, economic, and cultural information about a country—information often detailed in open sources.

1. Open source information provides a base for understanding classified materials. Despite large quantities of classified material produced by the IC, the amount of classified information produced on any one topic can be quite limited, and may be taken out of context if viewed only from a classified-source perspective. Perhaps the most important example today relates to terrorism, where open source information can fill gaps and create links that allow analysts to better understand fragmented intelligence, rumored terrorist plans, possible means of attack, and potential targets.

1. Open source materials can protect sources and methods. Sometimes an intelligence judgment that is actually informed with sensitive, classified information can be defended on the basis of open source reporting. This can prove useful when policymakers need to explain policy decisions or communicate with foreign officials without compromising classified sources.

1. Only open source can “store history.” A robust open source program can, in effect, gather data to monitor the world’s cultures and how they change with time. This is difficult, if not impossible, using the “snapshots” provided by classified collection methods. (The Commission on the Intelligence Capabilities, 378-379).

The United States National Commission on Terrorist Attacks Upon the United States issued a report in July 2004 recommending the creation of an open-source intelligence agency, but without further detail or comment. Subsequently, the WMD Commission (also known as the Robb-Silberman Commission) report in March 2005 recommended the creation of an Open Source Directorate at the CIA.

Both of these reports, while recent, are relatively superficial, and both ignore the decades of advocacy for a proper national focus on OSINT from 1988 to date. They also err, severely, in assuming that the same Central Intelligence Agency that has obstinantly refused to take open source information seriously for decades, should be charged with developing new capabilities that are totally outside its existing culture and cult of secrecy.

One possible solution, one being explored by the Chief Information Officer of the U.S. Intelligence Community, relates to open standards that allow for more sharing across organizational boundaries. A federated approach to OSINT in which each cabinet agency has its own OSINT capabilities, but shares requirements (to avoid redundant collection) and the fruits of the collection, via a global network, could eventually be integrated with Google, Amazon, and a wide variety of private sector aggregations of information. The emergence of the semantic web and synthetic information architecture, and of the Open Hypertextdocument System (OHS) invented by Doug Englebart, opens the way to both free synthetic answers distilled from massive aggregations, and for-fee footnotes or paragraphs at the micro-cash level.

Modern History of OSINT

In 1938 H. G. Well published "The World Brain" as a vision for the future that is emerging today, where the Internet makes it possible for a Wikipedia to emerge that not only has the potential to publish free useful knowledge on all topics, but also to bring together social networks of interested parties with varying levels of expertise. Since then a number of books have emerged including Pierre Teilhard de Chardin's "The Phenomenon of Man" with its concept of "noosphere," and more recently, books such as Howard Rheingold's "Smart Mobs," James Surowiecki's "Wisdom of the Crowds," and Glen Reynolds, "An Army of Davids."

The earliest known modern reference to the need for open source intelligence in the public service is by Quincy Wright, a pioneer in the international relations and political science field. His article, "Project for a World Intelligence Center", appeared in Conflict Resolution, Volume 1, Number 1 (1957), and inspired Robert Steele's original conceptualization of an Open Source Agency, as well a decade of reflection on peacekeeping intelligence, virtual diplomacy and information peacekeeping. It merits comment that Quincy Wright wrote his seminal article while associated with the Carnegie Endowment for International Peace, which remains a vibrant contributor to informed dialog. Other important institutions in this vein include the U.S. Institute for Peace, and the U.S. Army's Peacekeeping and Stability Operations Institute as well as its Strategic Studies Institute, both located within the Army War College. More recently, the Swedish Folke Bernadotte Academy had created, and opened to the public, the first structured PKI Course description.doc course in PKI Course Curriculum.doc peacekeeping intelligence.

Today the single most important source on the collective intelligence movement is the Co-Intelligence Institute founded by Tom Atlee, who is also the author of "The Tao of Democracy: Using Co-Intelligence to Create a World that Works for All'.

Within the secret world of intelligence, where other nations are often smarter about how to collect intelligence inexpensively, but where the USA, because of its massive budget, is the only power whose wisdom, if applied, could impact in very positive manner on the rest of the world, the OSINT war began in 1988, when the U.S. Marine Corps spent $20 million to create the Marine Corps Intelligence Command (then Center) and realized that 80% of the information it needed to support expeditionary policy, acquisition, and operations was not secret, not in English, not online, and not known to anyone in the National Capital Area. As with most major innovations seeking to reform massive bureaucracies, this will be a 25-year war with victory by the public expected in 2012-2013.

For a complete review of intelligence reform efforts from the inception of the U.S. Intelligence Community in 1947 to date, see the Congressional Research Service report. It merits comment that OSINT, empowered by the Internet, by collective intelligence, and by the new public appreciation for the value of sharing non-rival information to create added wealth, is the first truly significant "threat" to the secret world. Here to fore, secret failures were secret; secret waste was secret; and there was neither accountability nor rivalry. Now that OSINT is "the rival store" as described by Alvin and Heidi Toffler in "War and Anti-War," the secret world--at least in the USA where it is the least effective part of the military-industrial complex--is in decline.

In Fall 1992 Senator David Boren, then Chairman of the Senate Select Committee on Intelligence (SSCI), sponsored the National Security Act of 1992, attempting to achieve modest reform in the U.S. Intelligence Community. His counterpart on the House Permanent Select Committee on Intelligence (HPSCI) was Congressman McCurdy. The House version of the legislation included a separate Open Source Office, at the suggestion of Larry Prior, a Marine Reservist familiar with the MCIC experience and then serving on the HPSCI staff. This legislation was defeated by a combination of opposition from Senator John Warner of Virginia, who feared that reform would reduce intelligence jobs in Virginia (this fear is misplaced--every classified job that is eliminated can fund two or more unclassified positions), and a letter from then Secretary of Defense Dick Cheney that severely mis-represented the dangers to our national security of transferring the three *national* intelligence agencies out of the Department of Defense to a truly independent intelligence community. Within the past several years, LtGen Dr. Brent Scowcraft studied this issue again and repeated the recommendation that the three national agencies be removed from the Department of Defense, only to be dismissed by now Vice President Dick Cheney.

In August 1995 the Aspin-Brown Commission sponsored a benchmark exercise on Burundi in which a private sector party, with six telehone calls, overnight, completely crushed the entire U.S. Intelligence Community, producing the following:

-- From the Institute of Scientific Information (ISI), the top 100 international experts on Burundi available for immediate debriefing;

-- From LEXIS-NEXIS, the top international reporters on Burundi, available for immediate debriefing. -- From Oxford Analytica, twenty two-page reports on the political-military significance of Burundi at the Presidential, Prime Ministerial, and CEO levels;

-- From Jane's Information Group, one paragraph summaries of every article ever published about Burundi, and new, tailor-made tribal orders of battle created overnight for this need

-- From East View Cartographic, 1:50,000 meter combat charts with contour lines from Russia (the Americans do not have combat charts for 90% of the world--they rely too heavily on secret imagery satellites.

-- From SPOT Image in France, commercial imagery for Burundi, 100% coverage, cloud-free, less than three years old, in the archive and inexpensively available.

The CIA (which sponsors the Open Source Center today) had a small map of Africa and a regional economic study with flawed (Western) premises. There was no clandestine human intelligence, no imagery intelligence, and no signals intelligence, and of course no open source intelligence either. While there are differing accounts of the degree to which this exercise influenced the Commission, it appears to have been helpful to The Honorable Lee Hamilton, who later served on the 9-11 Commission.

Concurrent with the Commission's effort, the HPSCI commissioned its own review. Staff Director Mark Lowenthal, who led a masterful study called "IC21: The Intelligence Community in the 21st Century".

Several DCI's in succession refused to implement the Aspin-Brown Commission recommendations, including the recommendation in 1996 that stated that US access to open sources was "severely deficient" and that this should be a "top priority" for both funding and DCI attention. Consequently, the US entered the new century completely ignorant of all of the open source information in all languages relevant to identifying and containing Al Qaeda, which has been active, publicly, since 1988.

In July 1997 then DCI George Tenet received the report, "The Challenge of Global Coverage" conducted by Senior Intelligence Service officer Boyd Sutton. After interviewing virtually all of the Assistant Secretaries of Defense and State, and the heads of the varied intelligence organizations, the report recommended that the U.S. Intelligence Community, which obsesses on seven "hard targets" and continues to ignore the Third World and lower tier topics including the ten topics identified by the United Nations report of 2004, "Creating a More Secure World," spend $1.5 billion dollars a year on OSINT--$10M for each of 150 lower tier countries or topics where most of the instability was located, but which did not, to the denied area mindsets at the time, represent cataclysmic threats to the USA.

Two years after 9-11, through the heroic efforts of a Senior Intelligence Service officer serving on detail to the 9-11 Commission, and because The Honorable Lee Hamilton has witnessed the outcome of the Burundi competition, page 413 of the 9-11 Commission Report included a recommended Open Source Agency completely independent of the Central Intelligence Agency.

In December 2005, the Director of National Intelligence appointed Eliot A. Jardines as the Assistant Deputy Director of National Intelligence for Open Source to serve as the Intelligence Community's senior intelligence officer for open source and to provide strategy, guidance and oversight for the Community's open source activities. In November 2005, the Director of National Intelligence (DNI) announced the creation of the Open Source Center. The OSC’s functions include the collection, analysis and research, and training to facilitate government-wide access and use of OSIF. The OSC builds on the expertise of the CIA’s Foreign Broadcast Information Service (FBIS), which has provided the U.S. Government a broad range of open source products and services since 1941 (including world media content available to the public through World News Connection).

Unfortunately, Mr. Jardines has no program authority, no funding outside of $5M a year (another $20M a year funds the Open Source Center's Large Scale Internet Exploitation (LSIE) initiative), and he has no staff other than two rotational employees. While he oversees roughly $250M a year in "old" capabilities such as the Foreign Broadcast Information System (FBIS), he has not been given the authorities or funds needed to actually create a competent national open source intelligence capability. Until such time as there is an international Open Source Agency and network funded at no less than $1.5 billion a year (including $600M for commercial imagery raw source procurement), the U.S. Intelligence Community should be considered completely ineffective in this arena. Mr. Jardines, severely constrained by his bureaucratic circumstances, has been unable to harness the distributed knowledge of the Nation, nor to even propose guidelines for collaborative information development and sharing.

Several other nations maintain open source intelligence gathering operations, for example Australia's Office of National Assessments (ONA), the UK's BBC Monitoring Service (BBCM) and the Swiss Army GeneralStab. Collection and analysis tasks are allocated in different ways by different agencies: for example, the ONA is responsible for analysis and is an Australian government intelligence agency, while the BBCM is government-funded civilian organization concerned solely with collection and aggregation of news sources, and which employs civilian journalists. The "allied" nations tend to subordinate and disrespect OSINT, while the Third World nations have learned to rely on it, partly because that is all they can afford.

In a speech at the Council on Foreign Relations in February 2006, Secretary of Defense Donald Rumsfeld seems to have acknowledged the importance of open media as a component of national security in the information age. Secretary Rumsfeld's Undersecretary of Defense for Intelligence, Dr. Stephen Cambone, told the Security Affairs Support Association (SASA) in / January 2004 that he required nothing less than universal coverage in all languages, 24/7. It is Dr. Cambone, rather than the CIA, that has been forcefully addressing the urgent needs of the military and other agencies for OSINT in support of defense policy-making, defense acquisition, and defense operations.

Librarians, Researchers, and Open Source Officers

Librarians play a very special role in the OSINT world. Each of the eight communities of interest below has its own librarians. While Librarians come together at the Special Library Association, and independent information researchers at the Assocation of Independent Information Progressionals, or in secret gatherings within their respective secret communities, we still do not have a global conference or network that brings everyone together openly. It may be that such a new event is needed. In the meantime, Wikipedia 2006 is one place where cross-fertilization can take place.

Traditional librarians have begun to migrate away from the hard copy acquisition and internal classification roles, and emerged as very talented "scouts" able to discover, disciminate, distill, and disseminate essential information for their respective constituencies. When the World Brain emerges from the intersection of Wikipedia, Amazon, and Google, it will be librarians that will have the most important role to play in terms of providing structure and human-based linkage.

Librarians are encouraged to create their own sub-title page within their respective constituencies below.

Foreign Government OSINT

Contributions Eagerly Sought

The Republic of South Africa has pioneered the first truly continental early warning and open source information sharing system, and done so with a remarkably cost-effective and easy to proliferate approach that relies on open source software.

Military OSINT

Contributions Eagerly Sought

The single most important international pioneer in military OSINT is Research Arno Reuser, head of the Open Source Intelligence branch of the Dutch Military Intelligence organization. His deep knowledge of sources, softwares, and services, including his knowledge of fair and alternative pricing, is without peer. His latest briefing is available [1]here. His PERL scrips may become avaiable here shortly.

Law Enforcement OSINT

Contributions Eagerly Sought

The top OSINT practioners in the world within law enforcement are Scotland Yard, where the founder, Detective Steve Edwards, earned Honours as a Member of the British Empire (MBE) for his extraordinary work putting arms merchants and terrorists into jail faster, better, cheaper; and the Royal Canadian Mounted Police (RCMP), where Ms. Lori Schnitker pioneered low cost OSINT as a means of vetting and validating informants, and developing contextual leads for more traditional investigative methods.

Commercial Intelligence

Commercial Intelligence is not to be confused with Business Intelligence which is the term mis-uded by vendors of data mining capabilities and "dashboards." It is also more expansive than "Competitive" or "Competitor" Intelligence, as it includes the multiple environments within which businesses conceptualize and bring to market new capabilities. The gold standard in competitive intelligence is set by the Academy of Competitive Intelligence. This group also has a professional association, the Society of Competitive Intelligence Professionals.

Academic OSINT

Contributions Eagerly Sought

Some universities, notably The Institute for Intelligence Studies at Mercyhurst College, have developed truly extraordinary programs to teach the craft of intelligence as a substantive advance on traditional research methods. Traditional research focuses on specific information domains where new knowledge can be devised; the proven process of intelligence as taught by select academic institutions, and some private organizations such as the Academy of Competitive Intelligence, focuses more on decision support, inclusive of requirements definition, collection management, source discovery and validation, multi-source fusion, and compelling, timely, actionable presentation.

Non-Governmental Organization (NGO) OSINT

Contributions Eagerly Sought

The United Nations no longer treats the word "intelligence" as a "dirty word." In combination, the Brahimi Report, the service of MajGen Patrick Cammaert, Royal Netherlands Marines (as Military Advisor to the Secretary General), and the High-Level Threat Panel, have all led to an appreciation for the fact that decision-support also known as intelligence is essential to strategic mandates, operational force development, and tactical success.

Other organizations such as the International Committee of the Red Cross (ICRC), Doctors Without Borders, and Greenpeace, have established very effective and relatively low-cost global OSINT operations. The US Department of Defense has recognized the importance of information available from NGO's and in its Defense Science Board report on "Transitions to and from Hostilities" identifies NGOs, and open source information, as vital to the success of peacekeeping missions.

Mass, Niche, and Independent Media OSINT

Contributions Eagerly Sought

Investigative Journalism is a rarity in this era, and corporate owners and corporate advertisers increasingly subvert honest journalists and turn them into nothing more than "filler" around the advertisements. In the United States of America journalism has become especially corrupt, with mainstream newspapers refusing fully-funded advertisements against the lies of the Administration that led to the war on Iraq.

There are, however, some bright spots. Seymour Hersh at the New Yorker, Robert Kaplan at The Atlantic Monthly, and David Kaplan (no relation) at US News & World Report, are consistently effective in exposing fraud, waste, and abuse. From England, Greg Pabst reported on the theft of the Presidential election of 2000 in Florida through blatant disenfranchisement of over 35,000 predominantly black voters, but the Democratic Party lacked the spine to use this intelligence in a timely public manner.

Civil Society OSINT (Includes Advocacy Groups, Labor Unions, Religions)

Contributions Eagerly Sought

References

One-Page Link Guide to OSINT References - Covers history, context, process, practice, and tools

Business Week "The Power of Us"

Forbes "Reinventing Intelligence"

TIME "The New Craft of Intelligence"

NATO Open Source Intelligence Handbook (2001)

NATO Open Source Intelligence Reader (2002)

NATO Intelligence Exploitation of the Internet (2002)

OSS.Net - International public OSINT network and website, 30,000 pages,updated continuously.

Terms of Reference for Future of Intelligence - provides matrix for the war between secret and open intelligence proponents regarding funding, priorities, and methods.

Multinational Information Operations Center - Briefing on Moving to the Next Level

Information Operations: The Book - Briefing on Book that Takes IO to the Next Level

Annual IO/OSINT/PKI Conference - Annual Conference (since 1992) for Forward Thinkers (100 Scholarships)

IO: Putting the I Back Into Dime - Monograph from the Strategic Studies Institute of the US Army War College

The New Craft of Intelligence - Monograph from the Strategic Studies Institute of the US Army War College

Four Free Books Online ON INTEL, NEW CRAFT, IO, and PKI

External links (Vendors)

• BBC Monitoring Subscription service from the BBC.
• C4I.org- Private open-source intelligence clearinghouse.
• Dialog leader in providing online-based information services to organizations seeking competitive advantages.
• EastView Since 1989, East View has been the world's leader in providing high-quality information services from Russia and the NIS to the widest variety of customers.
• Factiva a Dow Jones & Reuters Company, provides world class global content, including Dow Jones and Reuters Newswires and The Wall Street Journal.
• Infosphere AB- Corporate OSINT from Commercial Intelligence & Knowledge Strategy consultancy Infosphere AB (Sweden).
• Lexis Nexis provides authoritative legal, news, public records and business information; including tax and regulatory publications in online, print or CD-ROM formats
• MarketResearch.com is the leading provider of global market intelligence products and services.
• Open Sources Center Subscription-based OSINT from ISRIA.
• The OSINT Center Geopolitical, Tourism and Energy OSINT from Alan Simpson
• SENTINEL- Government and private sector OSINT services from Elsag Solutions AG (Switzerland).
• Silobreaker- integrated sources and analytical webservice created by and for OSINT and CI users- Intelligence for Everyone (2005)
• World News Connection - global news from Federal Broadcast Information Service (FBIS is run by CIA, now under the DNI Open Source Center(OSC)).